Jul 23 17:55:42.474628 ip-10-0-142-147 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 23 17:55:42.474645 ip-10-0-142-147 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 23 17:55:42.474655 ip-10-0-142-147 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 23 17:55:42.475016 ip-10-0-142-147 systemd[1]: Failed to start Kubernetes Kubelet. Jul 23 17:55:52.586336 ip-10-0-142-147 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 23 17:55:52.586351 ip-10-0-142-147 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 4ebb97e78eda455bbcc8c30a72cdc415 -- Jul 23 17:57:30.429124 ip-10-0-142-147 systemd[1]: Starting Kubernetes Kubelet... Jul 23 17:57:30.872171 ip-10-0-142-147 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 17:57:30.872171 ip-10-0-142-147 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 17:57:30.872171 ip-10-0-142-147 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 17:57:30.872171 ip-10-0-142-147 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 17:57:30.872171 ip-10-0-142-147 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 17:57:30.873259 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.873050 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 17:57:30.880196 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880181 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:57:30.880196 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880196 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880200 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880203 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880207 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880210 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880213 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880216 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880218 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880221 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880224 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880227 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880235 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880237 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880240 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880243 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880246 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880248 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880251 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880253 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:57:30.880261 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880256 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880259 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880262 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880265 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880268 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880271 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880274 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880276 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880279 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880282 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880286 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880290 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880293 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880295 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880298 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880301 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880304 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880307 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880310 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:57:30.880692 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880314 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880317 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880320 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880322 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880325 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880327 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880330 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880332 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880334 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880337 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880339 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880342 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880344 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880346 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880349 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880352 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880355 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880358 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880360 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880363 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:57:30.881154 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880366 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880368 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880371 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880373 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880375 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880378 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880381 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880384 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880386 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880389 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880391 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880394 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880397 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880399 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880402 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880405 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880408 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880412 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880415 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:57:30.881614 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880418 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880420 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880423 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880425 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880428 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880431 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880433 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880436 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880439 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880442 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880445 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880447 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880450 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880452 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880456 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880459 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880462 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880464 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880467 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:57:30.882072 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880470 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.880473 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881153 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881160 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881163 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881166 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881169 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881173 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881175 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881178 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881181 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881184 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881186 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881189 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881192 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881194 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881197 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881200 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881202 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881205 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:57:30.882520 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881208 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881211 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881213 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881216 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881219 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881221 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881224 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881226 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881229 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881231 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881234 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881236 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881239 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881241 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881244 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881247 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881250 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881253 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881256 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881258 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:57:30.883012 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881263 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881267 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881270 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881273 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881276 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881279 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881283 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881286 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881289 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881291 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881294 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881296 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881299 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881302 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881304 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881307 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881310 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881312 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881315 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881317 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:57:30.883489 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881320 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881322 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881325 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881327 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881330 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881332 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881335 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881337 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881341 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881344 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881347 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881349 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881352 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881355 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881357 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881360 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881364 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881367 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881370 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:57:30.884040 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881373 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881376 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881378 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881381 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881383 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881386 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881389 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881392 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881395 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881397 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881400 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881403 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881405 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881408 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881411 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881413 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881416 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881419 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881421 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:57:30.884509 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881423 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881426 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881429 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881785 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881789 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881792 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881795 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881798 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881817 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881822 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881824 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881827 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881830 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881833 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881836 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881840 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881844 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881848 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881851 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:57:30.885236 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881854 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881857 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881860 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881863 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881865 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881868 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881870 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881873 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881876 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881879 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881881 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881884 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881887 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881890 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881893 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881895 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881898 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881901 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881903 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:57:30.885720 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881905 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881908 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881911 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881913 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881916 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881918 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881921 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881924 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881926 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881929 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881931 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881934 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881937 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881940 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881942 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881945 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881948 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881950 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881953 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881955 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:57:30.886285 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881958 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881960 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881963 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881965 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881968 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881970 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881972 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881975 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881977 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881980 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881982 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881985 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881988 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881991 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881993 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881996 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.881999 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882001 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882004 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882006 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:57:30.886822 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882009 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882011 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882014 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882017 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882020 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882024 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882026 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882029 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882031 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882034 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882036 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882039 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882042 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882044 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882047 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882049 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882052 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882054 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882057 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:57:30.887308 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882059 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882062 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882064 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882067 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882069 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882141 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882150 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882156 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882161 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882165 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882169 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882173 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882177 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882181 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882184 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882188 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882191 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882194 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882197 2564 flags.go:64] FLAG: --cgroup-root="" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882200 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882203 2564 flags.go:64] FLAG: --client-ca-file="" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882206 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882209 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 17:57:30.887751 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882213 2564 flags.go:64] FLAG: --cluster-domain="" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882216 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882219 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882222 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882225 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882229 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882232 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882234 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882238 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882240 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882243 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882246 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882249 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882252 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882255 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882260 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882263 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882265 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882268 2564 flags.go:64] FLAG: --enable-server="true" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882272 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882277 2564 flags.go:64] FLAG: --event-burst="100" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882280 2564 flags.go:64] FLAG: --event-qps="50" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882283 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882287 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 17:57:30.888354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882290 2564 flags.go:64] FLAG: --eviction-hard="" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882294 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882297 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882300 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882303 2564 flags.go:64] FLAG: --eviction-soft="" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882306 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882309 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882311 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882314 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882317 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882320 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882323 2564 flags.go:64] FLAG: --feature-gates="" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882327 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882330 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882332 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882336 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882339 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882342 2564 flags.go:64] FLAG: --help="false" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882345 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-142-147.ec2.internal" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882348 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882351 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882354 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882357 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 17:57:30.888998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882361 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882364 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882367 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882370 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882373 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882376 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882379 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882382 2564 flags.go:64] FLAG: --kube-reserved="" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882386 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882389 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882392 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882395 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882398 2564 flags.go:64] FLAG: --lock-file="" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882400 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882403 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882406 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882411 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882414 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882417 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882427 2564 flags.go:64] FLAG: --logging-format="text" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882431 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882434 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882437 2564 flags.go:64] FLAG: --manifest-url="" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882440 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882445 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 17:57:30.889568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882448 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882452 2564 flags.go:64] FLAG: --max-pods="110" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882455 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882458 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882461 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882464 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882467 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882471 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882474 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882481 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882484 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882487 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882490 2564 flags.go:64] FLAG: --pod-cidr="" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882493 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882498 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882501 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882506 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882509 2564 flags.go:64] FLAG: --port="10250" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882512 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882516 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07bb401656725e990" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882519 2564 flags.go:64] FLAG: --qos-reserved="" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882522 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882525 2564 flags.go:64] FLAG: --register-node="true" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882528 2564 flags.go:64] FLAG: --register-with-taints="" Jul 23 17:57:30.890199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882531 2564 flags.go:64] FLAG: --registry-burst="10" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882534 2564 flags.go:64] FLAG: --registry-qps="5" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882537 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882540 2564 flags.go:64] FLAG: --reserved-memory="" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882544 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882547 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882550 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882553 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882556 2564 flags.go:64] FLAG: --runonce="false" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882559 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882562 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882565 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882568 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882570 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882573 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882577 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882580 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882583 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882585 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882588 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882591 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882594 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882597 2564 flags.go:64] FLAG: --system-cgroups="" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882600 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882606 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 17:57:30.890795 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882609 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882612 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882616 2564 flags.go:64] FLAG: --tls-min-version="" Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882619 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882622 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882625 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882628 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882631 2564 flags.go:64] FLAG: --v="2" Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882635 2564 flags.go:64] FLAG: --version="false" Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882639 2564 flags.go:64] FLAG: --vmodule="" Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882642 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.882645 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882760 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882765 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882768 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882771 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882773 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882776 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882778 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882781 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882784 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882786 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882791 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:57:30.891415 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882793 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882796 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882798 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882813 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882816 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882819 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882821 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882825 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882829 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882832 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882835 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882837 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882840 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882842 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882845 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882847 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882850 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882853 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882855 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:57:30.892171 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882859 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882863 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882866 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882869 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882872 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882875 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882878 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882880 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882883 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882886 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882889 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882892 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882898 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882901 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882904 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882906 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882909 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882911 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882914 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882916 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:57:30.892973 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882919 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882924 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882927 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882930 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882932 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882935 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882937 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882942 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882945 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882948 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882951 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882954 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882957 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882960 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882962 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882965 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882968 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882970 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882973 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882975 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:57:30.893837 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882978 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882981 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882983 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882987 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882992 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882995 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.882998 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883001 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883003 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883006 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883008 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883011 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883013 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883018 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883021 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883024 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883027 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883029 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883032 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:57:30.894704 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883034 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883037 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883039 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883042 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883045 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883047 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883050 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883053 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883055 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.883058 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.883765 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.894097 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.894119 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.894139 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.894148 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 17:57:30.895505 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894231 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894240 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894245 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894250 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894256 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894261 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894267 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894271 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894276 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894280 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894285 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894290 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894294 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894298 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894303 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894307 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894311 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894315 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894320 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:57:30.896004 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894324 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894328 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894335 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894341 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894347 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894351 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894356 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894360 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894365 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894370 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894375 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894380 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894384 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894389 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894393 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894398 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894402 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894407 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894411 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894416 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:57:30.896518 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894430 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894435 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894439 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894443 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894447 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894451 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894456 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894460 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894464 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894468 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894473 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894477 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894481 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894485 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894489 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894494 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894499 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894503 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894507 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894511 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:57:30.897334 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894515 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894519 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894525 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894529 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894533 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894537 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894541 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894547 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894554 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894559 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894563 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894567 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894572 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894576 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894580 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894584 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894588 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894593 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894597 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:57:30.898070 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894603 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894609 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894614 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894618 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894623 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894627 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894632 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894638 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894643 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894647 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894651 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894655 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894659 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894663 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894668 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894673 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894677 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894681 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894686 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894690 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:57:30.899311 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894694 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.894701 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894902 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894912 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894917 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894922 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894927 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894932 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894936 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894941 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894945 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894949 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894953 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894959 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894963 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:57:30.899833 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894967 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894971 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894976 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894980 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894984 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894988 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894992 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.894996 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895001 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895005 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895009 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895014 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895018 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895023 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895027 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895031 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895035 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895039 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895044 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895048 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:57:30.900201 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895052 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895056 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895060 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895064 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895071 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895078 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895084 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895089 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895094 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895098 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895102 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895108 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895112 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895118 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895123 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895128 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895132 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895136 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895140 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895145 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:57:30.900730 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895149 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895153 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895157 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895162 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895166 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895170 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895175 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895179 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895183 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895187 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895191 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895195 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895200 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895204 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895208 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895213 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895217 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895221 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895226 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:57:30.901286 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895230 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895236 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895242 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895247 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895251 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895257 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895261 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895265 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895269 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895274 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895278 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895282 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895286 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895290 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895294 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895298 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895302 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895306 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895310 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895315 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:57:30.901750 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895319 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:57:30.902260 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895323 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:57:30.902260 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895330 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:57:30.902260 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895335 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:57:30.902260 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895340 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:57:30.902260 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895345 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:57:30.902260 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:30.895350 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:57:30.902260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.895358 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 17:57:30.902260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.895558 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 17:57:30.902260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.898642 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 17:57:30.902260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.900245 2564 server.go:1019] "Starting client certificate rotation" Jul 23 17:57:30.902260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.900341 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 17:57:30.902260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.900407 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 17:57:30.903391 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.903373 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 17:57:30.904961 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.904944 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 17:57:30.919563 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.919545 2564 log.go:25] "Validated CRI v1 runtime API" Jul 23 17:57:30.924474 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.924455 2564 log.go:25] "Validated CRI v1 image API" Jul 23 17:57:30.925315 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.925298 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 17:57:30.930513 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.930481 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9c2c9052-40bc-4a5c-aec7-0db689af8317:/dev/nvme0n1p4 f11f9d37-6b1f-4b19-9190-7c0639563996:/dev/nvme0n1p3] Jul 23 17:57:30.930513 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.930503 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 17:57:30.935932 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.935913 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 17:57:30.936823 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.936694 2564 manager.go:216] Machine: {Timestamp:2026-07-23 17:57:30.935261068 +0000 UTC m=+0.376759654 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099620 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21874172c9d7ff62e636aae60ad959 SystemUUID:ec218741-72c9-d7ff-62e6-36aae60ad959 BootID:4ebb97e7-8eda-455b-bcc8-c30a72cdc415 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a8:32:22:c1:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a8:32:22:c1:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:96:89:43:8e:a3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 17:57:30.936823 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.936819 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 17:57:30.936951 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.936939 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 17:57:30.940433 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.940407 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 17:57:30.940574 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.940436 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-147.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 17:57:30.942050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.942040 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 17:57:30.942084 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.942054 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 17:57:30.942084 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.942061 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 17:57:30.942823 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.942813 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 17:57:30.944715 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.944704 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 17:57:30.948311 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.948300 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 23 17:57:30.948432 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.948423 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 17:57:30.948469 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.948454 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 23 17:57:30.948469 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.948467 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 17:57:30.948521 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.948481 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 17:57:30.948521 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.948490 2564 kubelet.go:398] "Adding apiserver pod source" Jul 23 17:57:30.948521 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.948499 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 17:57:30.948905 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.948893 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 17:57:30.948952 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.948910 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 17:57:30.951528 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.951510 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 17:57:30.953883 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.953866 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 17:57:30.953975 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.953897 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 17:57:30.955192 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.955177 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 17:57:30.955260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.955196 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 17:57:30.955260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.955203 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 17:57:30.955260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.955208 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 17:57:30.955260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.955216 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 17:57:30.955260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.955222 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 17:57:30.955260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.955230 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 17:57:30.955260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.955235 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 17:57:30.955260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.955243 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 17:57:30.955260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.955248 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 17:57:30.955260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.955257 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 17:57:30.955260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.955262 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 17:57:30.956158 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.956148 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 17:57:30.956158 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.956157 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 17:57:30.959426 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.959401 2564 server.go:1268] "Started kubelet" Jul 23 17:57:30.959586 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.959522 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 17:57:30.959642 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.959606 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 17:57:30.959681 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.959668 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 17:57:30.960129 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.960107 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 17:57:30.960262 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:30.960150 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 17:57:30.960262 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:30.960186 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 17:57:30.960305 ip-10-0-142-147 systemd[1]: Started Kubernetes Kubelet. Jul 23 17:57:30.962452 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.962434 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 23 17:57:30.962631 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.962608 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 17:57:30.964998 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:30.964968 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 17:57:30.968790 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:30.967697 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-147.ec2.internal.18c4fcede116d001 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-147.ec2.internal,UID:ip-10-0-142-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-147.ec2.internal,},FirstTimestamp:2026-07-23 17:57:30.959376385 +0000 UTC m=+0.400874970,LastTimestamp:2026-07-23 17:57:30.959376385 +0000 UTC m=+0.400874970,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-147.ec2.internal,}" Jul 23 17:57:30.972135 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.972113 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 17:57:30.972220 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.972122 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 17:57:30.972271 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:30.972237 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-147.ec2.internal\" not found" Jul 23 17:57:30.972920 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.972900 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 17:57:30.973000 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.972924 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 17:57:30.973000 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.972970 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 17:57:30.973094 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.972935 2564 factory.go:55] Registering systemd factory Jul 23 17:57:30.973094 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.973048 2564 factory.go:223] Registration of the systemd container factory successfully Jul 23 17:57:30.973171 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.973126 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 23 17:57:30.973171 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.973132 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 23 17:57:30.973374 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.973355 2564 factory.go:153] Registering CRI-O factory Jul 23 17:57:30.973374 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.973371 2564 factory.go:223] Registration of the crio container factory successfully Jul 23 17:57:30.973483 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.973416 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 17:57:30.973483 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.973436 2564 factory.go:103] Registering Raw factory Jul 23 17:57:30.973483 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.973446 2564 manager.go:1196] Started watching for new ooms in manager Jul 23 17:57:30.973932 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.973918 2564 manager.go:349] Starting recovery of all containers Jul 23 17:57:30.975503 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:30.975476 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 17:57:30.975641 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:30.975620 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 17:57:30.977210 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.977186 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6qpw8" Jul 23 17:57:30.984609 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.984596 2564 manager.go:354] Recovery completed Jul 23 17:57:30.987980 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.987958 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6qpw8" Jul 23 17:57:30.989816 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.989789 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:57:30.991615 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.991598 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:57:30.991680 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.991629 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:57:30.991680 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.991645 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:57:30.992296 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.992281 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:57:30.992681 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.992663 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 17:57:30.992681 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.992678 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 17:57:30.992791 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.992695 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 23 17:57:30.993048 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.993035 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:57:30.993096 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.993060 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:57:30.993096 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.993080 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:57:30.993817 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:30.993695 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-147.ec2.internal.18c4fcede302b60d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-147.ec2.internal,UID:ip-10-0-142-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-147.ec2.internal,},FirstTimestamp:2026-07-23 17:57:30.991613453 +0000 UTC m=+0.433112037,LastTimestamp:2026-07-23 17:57:30.991613453 +0000 UTC m=+0.433112037,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-147.ec2.internal,}" Jul 23 17:57:30.993911 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.993824 2564 policy_none.go:49] "None policy: Start" Jul 23 17:57:30.993911 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.993841 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 17:57:30.993911 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.993856 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 17:57:30.994915 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:30.994901 2564 policy_none.go:47] "Start" Jul 23 17:57:31.023405 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.023389 2564 manager.go:337] "Starting Device Plugin manager" Jul 23 17:57:31.035938 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.023425 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 17:57:31.035938 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.023438 2564 server.go:87] "Starting device plugin registration server" Jul 23 17:57:31.035938 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.023674 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 17:57:31.035938 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.023686 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 17:57:31.035938 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.023770 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 17:57:31.035938 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.023874 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 17:57:31.035938 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.023883 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 17:57:31.035938 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.025782 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 17:57:31.035938 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.025845 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-147.ec2.internal\" not found" Jul 23 17:57:31.109782 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.109746 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 17:57:31.110966 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.110948 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 17:57:31.111069 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.110987 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 17:57:31.111069 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.111011 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 17:57:31.111069 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.111053 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 17:57:31.113585 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.113563 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:57:31.124565 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.124500 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:57:31.125355 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.125338 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:57:31.125428 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.125367 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:57:31.125428 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.125381 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:57:31.125428 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.125402 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.138912 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.138890 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.139002 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.138917 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-147.ec2.internal\": node \"ip-10-0-142-147.ec2.internal\" not found" Jul 23 17:57:31.165152 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.165124 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-147.ec2.internal\" not found" Jul 23 17:57:31.211345 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.211314 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-147.ec2.internal"] Jul 23 17:57:31.211483 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.211405 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:57:31.212906 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.212889 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:57:31.213002 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.212922 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:57:31.213002 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.212935 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:57:31.214370 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.214355 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:57:31.214490 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.214468 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.214538 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.214500 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:57:31.215130 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.215111 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:57:31.215208 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.215144 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:57:31.215208 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.215155 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:57:31.215208 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.215111 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:57:31.215346 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.215222 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:57:31.215346 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.215238 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:57:31.216324 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.216311 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.216384 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.216335 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:57:31.217024 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.217001 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:57:31.217106 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.217039 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:57:31.217106 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.217052 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:57:31.236330 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.236316 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-147.ec2.internal\" not found" node="ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.240263 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.240246 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-147.ec2.internal\" not found" node="ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.266160 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.266141 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-147.ec2.internal\" not found" Jul 23 17:57:31.274930 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.274911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a43e806b4b4d8f64fbb04742acee7cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal\" (UID: \"1a43e806b4b4d8f64fbb04742acee7cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.275010 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.274938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a43e806b4b4d8f64fbb04742acee7cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal\" (UID: \"1a43e806b4b4d8f64fbb04742acee7cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.275010 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.274958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a44435e6f3df5d50b47d277e77112d1-config\") pod \"kube-apiserver-proxy-ip-10-0-142-147.ec2.internal\" (UID: \"4a44435e6f3df5d50b47d277e77112d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.366601 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.366572 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-147.ec2.internal\" not found" Jul 23 17:57:31.376010 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.375954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a43e806b4b4d8f64fbb04742acee7cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal\" (UID: \"1a43e806b4b4d8f64fbb04742acee7cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.376010 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.375988 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a43e806b4b4d8f64fbb04742acee7cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal\" (UID: \"1a43e806b4b4d8f64fbb04742acee7cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.376010 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.376006 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a44435e6f3df5d50b47d277e77112d1-config\") pod \"kube-apiserver-proxy-ip-10-0-142-147.ec2.internal\" (UID: \"4a44435e6f3df5d50b47d277e77112d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.376158 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.376035 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a44435e6f3df5d50b47d277e77112d1-config\") pod \"kube-apiserver-proxy-ip-10-0-142-147.ec2.internal\" (UID: \"4a44435e6f3df5d50b47d277e77112d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.376158 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.376066 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a43e806b4b4d8f64fbb04742acee7cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal\" (UID: \"1a43e806b4b4d8f64fbb04742acee7cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.376158 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.376067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a43e806b4b4d8f64fbb04742acee7cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal\" (UID: \"1a43e806b4b4d8f64fbb04742acee7cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.467330 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.467289 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-147.ec2.internal\" not found" Jul 23 17:57:31.539891 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.539864 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.544119 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.543516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.568135 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.568105 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-147.ec2.internal\" not found" Jul 23 17:57:31.674492 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.674394 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-147.ec2.internal\" not found" Jul 23 17:57:31.775192 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.775156 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-147.ec2.internal\" not found" Jul 23 17:57:31.876312 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.876287 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-147.ec2.internal\" not found" Jul 23 17:57:31.900334 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.900310 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 17:57:31.900509 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.900487 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 17:57:31.932285 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.932215 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:57:31.949709 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.949688 2564 apiserver.go:52] "Watching apiserver" Jul 23 17:57:31.967922 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.967896 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 17:57:31.968266 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.968245 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-x82j5","openshift-multus/multus-additional-cni-plugins-7zjxq","openshift-multus/multus-psq9w","openshift-multus/network-metrics-daemon-lg9sf","openshift-network-operator/iptables-alerter-8kbwh","kube-system/konnectivity-agent-9rmwf","openshift-dns/node-resolver-gwvmh","openshift-network-diagnostics/network-check-target-tcz9v","openshift-ovn-kubernetes/ovnkube-node-rfsgl","kube-system/global-pull-secret-syncer-865dp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b","openshift-cluster-node-tuning-operator/tuned-m24k7"] Jul 23 17:57:31.970000 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.969985 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:31.971669 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.971648 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.972871 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.972852 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal" Jul 23 17:57:31.972871 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.972867 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 17:57:31.972988 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.972950 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:31.973040 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.973018 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:57:31.974170 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.974156 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8kbwh" Jul 23 17:57:31.975460 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.975445 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9rmwf" Jul 23 17:57:31.976626 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.976607 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gwvmh" Jul 23 17:57:31.977959 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.977939 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:31.978032 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.978009 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:57:31.979365 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed871313-07b6-4f58-aaed-314636ffcc3d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:31.979459 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979380 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed871313-07b6-4f58-aaed-314636ffcc3d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:31.979459 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979423 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pxr6\" (UniqueName: \"kubernetes.io/projected/ed871313-07b6-4f58-aaed-314636ffcc3d-kube-api-access-6pxr6\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:31.979605 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979455 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-os-release\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.979605 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979479 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c385ce53-b210-4737-ae2b-a26aed9b4399-cni-binary-copy\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.979605 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-multus-socket-dir-parent\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.979605 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979533 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-run-netns\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.979605 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-var-lib-cni-bin\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.979781 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979617 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-etc-kubernetes\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.979781 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m4g7\" (UniqueName: \"kubernetes.io/projected/c385ce53-b210-4737-ae2b-a26aed9b4399-kube-api-access-4m4g7\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.979781 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979664 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhr2x\" (UniqueName: \"kubernetes.io/projected/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-kube-api-access-dhr2x\") pod \"network-metrics-daemon-lg9sf\" (UID: \"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26\") " pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:31.979781 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xtcp\" (UniqueName: \"kubernetes.io/projected/d04a87a4-9a0b-4840-8642-82500e37d420-kube-api-access-4xtcp\") pod \"iptables-alerter-8kbwh\" (UID: \"d04a87a4-9a0b-4840-8642-82500e37d420\") " pod="openshift-network-operator/iptables-alerter-8kbwh" Jul 23 17:57:31.979974 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed871313-07b6-4f58-aaed-314636ffcc3d-system-cni-dir\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:31.979974 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979719 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:31.979974 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979837 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c385ce53-b210-4737-ae2b-a26aed9b4399-multus-daemon-config\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.979974 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979918 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed871313-07b6-4f58-aaed-314636ffcc3d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:31.979974 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-hostroot\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.979974 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed871313-07b6-4f58-aaed-314636ffcc3d-cnibin\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:31.980269 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.979997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-run-multus-certs\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.980269 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.980020 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-system-cni-dir\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.980269 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.980045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-cnibin\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.980269 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.980070 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-run-k8s-cni-cncf-io\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.980269 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.980093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d04a87a4-9a0b-4840-8642-82500e37d420-iptables-alerter-script\") pod \"iptables-alerter-8kbwh\" (UID: \"d04a87a4-9a0b-4840-8642-82500e37d420\") " pod="openshift-network-operator/iptables-alerter-8kbwh" Jul 23 17:57:31.980269 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.980124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-multus-cni-dir\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.980269 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.980139 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-var-lib-cni-multus\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.980269 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.980154 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-var-lib-kubelet\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.980269 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.980195 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-multus-conf-dir\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:31.980269 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.980211 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs\") pod \"network-metrics-daemon-lg9sf\" (UID: \"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26\") " pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:31.980269 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.980227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d04a87a4-9a0b-4840-8642-82500e37d420-host-slash\") pod \"iptables-alerter-8kbwh\" (UID: \"d04a87a4-9a0b-4840-8642-82500e37d420\") " pod="openshift-network-operator/iptables-alerter-8kbwh" Jul 23 17:57:31.980269 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.980243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed871313-07b6-4f58-aaed-314636ffcc3d-os-release\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:31.980269 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.980262 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed871313-07b6-4f58-aaed-314636ffcc3d-cni-binary-copy\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:31.981257 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.981240 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:31.982674 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.982654 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:31.984341 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.984320 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x82j5" Jul 23 17:57:31.984761 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.984742 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 17:57:31.985743 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.985724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:31.985873 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.985751 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 17:57:31.985873 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.985765 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 17:57:31.987748 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:31.986197 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:57:31.987748 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.986636 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 17:57:31.987748 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.987035 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 17:57:31.987748 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.986640 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 17:57:31.988231 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.988188 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 17:57:31.988922 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.988488 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 17:57:31.988922 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.988506 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 17:57:31.989025 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.988993 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2c8v8\"" Jul 23 17:57:31.989757 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.989467 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6c7n8\"" Jul 23 17:57:31.990971 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.990585 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 17:57:31.990971 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.990611 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 17:57:31.990971 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.990620 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 17:57:31.990971 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.990629 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 17:57:31.990971 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.990669 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 17:57:31.990971 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.990622 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 17:57:31.990971 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.990617 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-c6z4k\"" Jul 23 17:57:31.990971 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.990754 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zz257\"" Jul 23 17:57:31.990971 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.990597 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 17:57:31.990971 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.990619 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9qpgt\"" Jul 23 17:57:31.990971 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.990842 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4g46k\"" Jul 23 17:57:31.990971 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.990848 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 17:57:31.990971 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.990778 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zngnw\"" Jul 23 17:57:31.990971 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.990915 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 17:57:31.991693 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.991157 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 17:57:31.991693 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.991287 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 17:57:31.991693 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.991329 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 17:57:31.991693 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.991338 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 17:57:31.991693 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.991376 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 17:57:31.991693 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.991395 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 17:57:31.991693 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.991331 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 17:57:31.991693 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.991534 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jwxqh\"" Jul 23 17:57:31.991693 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.991566 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 17:57:31.991693 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.991596 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7kz72\"" Jul 23 17:57:31.993007 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.992960 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 17:52:31 +0000 UTC" deadline="2028-04-17 00:11:37.172945502 +0000 UTC" Jul 23 17:57:31.993007 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.993006 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15198h14m5.179941934s" Jul 23 17:57:31.999405 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:31.999383 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 17:57:32.006550 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.006521 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal"] Jul 23 17:57:32.006798 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.006778 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 17:57:32.006923 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.006857 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-147.ec2.internal" Jul 23 17:57:32.010753 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.010732 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 17:57:32.017248 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.017229 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-147.ec2.internal"] Jul 23 17:57:32.017807 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.017792 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 17:57:32.040284 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.040072 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x6pfz" Jul 23 17:57:32.048403 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.048375 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x6pfz" Jul 23 17:57:32.074433 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.074414 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 17:57:32.080425 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-multus-conf-dir\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.080563 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080441 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-sysctl-conf\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.080563 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080466 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-run-netns\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.080563 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080491 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-env-overrides\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.080563 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080514 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-multus-conf-dir\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.080563 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed871313-07b6-4f58-aaed-314636ffcc3d-os-release\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.080864 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080604 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed871313-07b6-4f58-aaed-314636ffcc3d-os-release\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.080864 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed871313-07b6-4f58-aaed-314636ffcc3d-cni-binary-copy\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.080864 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed871313-07b6-4f58-aaed-314636ffcc3d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.080864 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-multus-socket-dir-parent\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.080864 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-systemd-units\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.080864 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080759 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-run-ovn\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.080864 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080786 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-run-netns\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.080864 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-etc-kubernetes\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.080864 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080837 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhr2x\" (UniqueName: \"kubernetes.io/projected/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-kube-api-access-dhr2x\") pod \"network-metrics-daemon-lg9sf\" (UID: \"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26\") " pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080891 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-multus-socket-dir-parent\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080910 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xtcp\" (UniqueName: \"kubernetes.io/projected/d04a87a4-9a0b-4840-8642-82500e37d420-kube-api-access-4xtcp\") pod \"iptables-alerter-8kbwh\" (UID: \"d04a87a4-9a0b-4840-8642-82500e37d420\") " pod="openshift-network-operator/iptables-alerter-8kbwh" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080939 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-etc-kubernetes\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080851 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed871313-07b6-4f58-aaed-314636ffcc3d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.080940 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-etc-openvswitch\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6098df05-0cf3-46ae-83ea-f16c1edf1260-host\") pod \"node-ca-x82j5\" (UID: \"6098df05-0cf3-46ae-83ea-f16c1edf1260\") " pod="openshift-image-registry/node-ca-x82j5" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c385ce53-b210-4737-ae2b-a26aed9b4399-multus-daemon-config\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081087 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-node-log\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-run-ovn-kubernetes\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-run-netns\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081239 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed871313-07b6-4f58-aaed-314636ffcc3d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-hostroot\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081329 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed871313-07b6-4f58-aaed-314636ffcc3d-cni-binary-copy\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081340 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-var-lib-openvswitch\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.081411 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-run-openvswitch\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-ovn-node-metrics-cert\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081410 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-hostroot\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081426 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d669e26-a0e5-4073-8f2c-64b52d9f959d-kubelet-config\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081482 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-sys-fs\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdk4\" (UniqueName: \"kubernetes.io/projected/cccf0c11-35a3-4785-8723-5f3c8002e4b7-kube-api-access-jmdk4\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-run-multus-certs\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081580 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e2c7a4c9-5c1f-45df-a280-ed8a43b90a93-konnectivity-ca\") pod \"konnectivity-agent-9rmwf\" (UID: \"e2c7a4c9-5c1f-45df-a280-ed8a43b90a93\") " pod="kube-system/konnectivity-agent-9rmwf" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081605 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-system-cni-dir\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-cnibin\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-run-k8s-cni-cncf-io\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081679 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c385ce53-b210-4737-ae2b-a26aed9b4399-multus-daemon-config\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081688 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-host\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081662 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-run-multus-certs\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081711 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-cnibin\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081715 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9frz\" (UniqueName: \"kubernetes.io/projected/9a34406b-fbaf-4512-871f-768a1a43ddc3-kube-api-access-t9frz\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081749 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-system-cni-dir\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.082198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081755 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-ovnkube-config\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed871313-07b6-4f58-aaed-314636ffcc3d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081759 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-run-k8s-cni-cncf-io\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081826 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x6w5\" (UniqueName: \"kubernetes.io/projected/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-kube-api-access-7x6w5\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081857 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d669e26-a0e5-4073-8f2c-64b52d9f959d-dbus\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d04a87a4-9a0b-4840-8642-82500e37d420-iptables-alerter-script\") pod \"iptables-alerter-8kbwh\" (UID: \"d04a87a4-9a0b-4840-8642-82500e37d420\") " pod="openshift-network-operator/iptables-alerter-8kbwh" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081909 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-lib-modules\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081934 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-tuned\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.081990 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e2c7a4c9-5c1f-45df-a280-ed8a43b90a93-agent-certs\") pod \"konnectivity-agent-9rmwf\" (UID: \"e2c7a4c9-5c1f-45df-a280-ed8a43b90a93\") " pod="kube-system/konnectivity-agent-9rmwf" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082017 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-run-systemd\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-socket-dir\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082088 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-device-dir\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-multus-cni-dir\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082142 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-var-lib-cni-multus\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-var-lib-kubelet\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs\") pod \"network-metrics-daemon-lg9sf\" (UID: \"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26\") " pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082232 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-multus-cni-dir\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.083167 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082241 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-var-lib-cni-multus\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d04a87a4-9a0b-4840-8642-82500e37d420-host-slash\") pod \"iptables-alerter-8kbwh\" (UID: \"d04a87a4-9a0b-4840-8642-82500e37d420\") " pod="openshift-network-operator/iptables-alerter-8kbwh" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082294 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-sysconfig\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082300 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-var-lib-kubelet\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082312 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-kubernetes\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082347 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d04a87a4-9a0b-4840-8642-82500e37d420-host-slash\") pod \"iptables-alerter-8kbwh\" (UID: \"d04a87a4-9a0b-4840-8642-82500e37d420\") " pod="openshift-network-operator/iptables-alerter-8kbwh" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.082352 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed871313-07b6-4f58-aaed-314636ffcc3d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.082690 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs podName:b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26 nodeName:}" failed. No retries permitted until 2026-07-23 17:57:32.582659127 +0000 UTC m=+2.024157715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs") pod "network-metrics-daemon-lg9sf" (UID: "b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082695 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d04a87a4-9a0b-4840-8642-82500e37d420-iptables-alerter-script\") pod \"iptables-alerter-8kbwh\" (UID: \"d04a87a4-9a0b-4840-8642-82500e37d420\") " pod="openshift-network-operator/iptables-alerter-8kbwh" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pxr6\" (UniqueName: \"kubernetes.io/projected/ed871313-07b6-4f58-aaed-314636ffcc3d-kube-api-access-6pxr6\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.082917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-os-release\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083053 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c385ce53-b210-4737-ae2b-a26aed9b4399-cni-binary-copy\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083057 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-os-release\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083157 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-var-lib-cni-bin\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083243 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c385ce53-b210-4737-ae2b-a26aed9b4399-host-var-lib-cni-bin\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083301 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-kubelet\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.084050 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-log-socket\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-cni-netd\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m4g7\" (UniqueName: \"kubernetes.io/projected/c385ce53-b210-4737-ae2b-a26aed9b4399-kube-api-access-4m4g7\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-systemd\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083410 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-sys\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad-tmp-dir\") pod \"node-resolver-gwvmh\" (UID: \"d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad\") " pod="openshift-dns/node-resolver-gwvmh" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xq4r\" (UniqueName: \"kubernetes.io/projected/6098df05-0cf3-46ae-83ea-f16c1edf1260-kube-api-access-8xq4r\") pod \"node-ca-x82j5\" (UID: \"6098df05-0cf3-46ae-83ea-f16c1edf1260\") " pod="openshift-image-registry/node-ca-x82j5" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083471 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhjf7\" (UniqueName: \"kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7\") pod \"network-check-target-tcz9v\" (UID: \"d3acdd92-76f4-4cf9-b91e-4ba8016f441a\") " pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083491 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083512 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed871313-07b6-4f58-aaed-314636ffcc3d-system-cni-dir\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083541 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-slash\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083558 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083579 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-sysctl-d\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083599 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a34406b-fbaf-4512-871f-768a1a43ddc3-tmp\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4s7k\" (UniqueName: \"kubernetes.io/projected/d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad-kube-api-access-s4s7k\") pod \"node-resolver-gwvmh\" (UID: \"d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad\") " pod="openshift-dns/node-resolver-gwvmh" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083642 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-ovnkube-script-lib\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed871313-07b6-4f58-aaed-314636ffcc3d-cnibin\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.084875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-run\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.085696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-cni-bin\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.085696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083718 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6098df05-0cf3-46ae-83ea-f16c1edf1260-serviceca\") pod \"node-ca-x82j5\" (UID: \"6098df05-0cf3-46ae-83ea-f16c1edf1260\") " pod="openshift-image-registry/node-ca-x82j5" Jul 23 17:57:32.085696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083742 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-registration-dir\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.085696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083744 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c385ce53-b210-4737-ae2b-a26aed9b4399-cni-binary-copy\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.085696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083760 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-var-lib-kubelet\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.085696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-modprobe-d\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.085696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.083777 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed871313-07b6-4f58-aaed-314636ffcc3d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.085696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.084154 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad-hosts-file\") pod \"node-resolver-gwvmh\" (UID: \"d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad\") " pod="openshift-dns/node-resolver-gwvmh" Jul 23 17:57:32.085696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.084535 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed871313-07b6-4f58-aaed-314636ffcc3d-cnibin\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.085696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.084617 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed871313-07b6-4f58-aaed-314636ffcc3d-system-cni-dir\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.090452 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.090428 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 17:57:32.094199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.094176 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xtcp\" (UniqueName: \"kubernetes.io/projected/d04a87a4-9a0b-4840-8642-82500e37d420-kube-api-access-4xtcp\") pod \"iptables-alerter-8kbwh\" (UID: \"d04a87a4-9a0b-4840-8642-82500e37d420\") " pod="openshift-network-operator/iptables-alerter-8kbwh" Jul 23 17:57:32.094279 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.094212 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhr2x\" (UniqueName: \"kubernetes.io/projected/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-kube-api-access-dhr2x\") pod \"network-metrics-daemon-lg9sf\" (UID: \"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26\") " pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:32.103845 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.103805 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pxr6\" (UniqueName: \"kubernetes.io/projected/ed871313-07b6-4f58-aaed-314636ffcc3d-kube-api-access-6pxr6\") pod \"multus-additional-cni-plugins-7zjxq\" (UID: \"ed871313-07b6-4f58-aaed-314636ffcc3d\") " pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.106171 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.106150 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m4g7\" (UniqueName: \"kubernetes.io/projected/c385ce53-b210-4737-ae2b-a26aed9b4399-kube-api-access-4m4g7\") pod \"multus-psq9w\" (UID: \"c385ce53-b210-4737-ae2b-a26aed9b4399\") " pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.142395 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:32.142362 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a43e806b4b4d8f64fbb04742acee7cb.slice/crio-483c1de3cb33dc6974aaf61a853c618330eddaa75652f7d832d1c55b1c8aee20 WatchSource:0}: Error finding container 483c1de3cb33dc6974aaf61a853c618330eddaa75652f7d832d1c55b1c8aee20: Status 404 returned error can't find the container with id 483c1de3cb33dc6974aaf61a853c618330eddaa75652f7d832d1c55b1c8aee20 Jul 23 17:57:32.142950 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:32.142921 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a44435e6f3df5d50b47d277e77112d1.slice/crio-bfd22fbe9928f0078f3d0d89fc41f7f26d46a1987a5b656163131c8c9f76ee23 WatchSource:0}: Error finding container bfd22fbe9928f0078f3d0d89fc41f7f26d46a1987a5b656163131c8c9f76ee23: Status 404 returned error can't find the container with id bfd22fbe9928f0078f3d0d89fc41f7f26d46a1987a5b656163131c8c9f76ee23 Jul 23 17:57:32.147580 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.147565 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 17:57:32.185069 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.184997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-modprobe-d\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.185069 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185026 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad-hosts-file\") pod \"node-resolver-gwvmh\" (UID: \"d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad\") " pod="openshift-dns/node-resolver-gwvmh" Jul 23 17:57:32.185069 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-sysctl-conf\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.185069 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-run-netns\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185103 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-run-netns\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185124 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad-hosts-file\") pod \"node-resolver-gwvmh\" (UID: \"d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad\") " pod="openshift-dns/node-resolver-gwvmh" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-env-overrides\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185162 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-modprobe-d\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185193 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-systemd-units\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-run-ovn\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-etc-openvswitch\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185237 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-sysctl-conf\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6098df05-0cf3-46ae-83ea-f16c1edf1260-host\") pod \"node-ca-x82j5\" (UID: \"6098df05-0cf3-46ae-83ea-f16c1edf1260\") " pod="openshift-image-registry/node-ca-x82j5" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185282 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-run-ovn\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-node-log\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185305 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6098df05-0cf3-46ae-83ea-f16c1edf1260-host\") pod \"node-ca-x82j5\" (UID: \"6098df05-0cf3-46ae-83ea-f16c1edf1260\") " pod="openshift-image-registry/node-ca-x82j5" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185323 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-node-log\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185331 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-run-ovn-kubernetes\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185349 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-etc-openvswitch\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.185354 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185362 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-systemd-units\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185360 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185397 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185392 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-run-ovn-kubernetes\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185400 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.185473 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185483 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-var-lib-openvswitch\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.185512 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret podName:1d669e26-a0e5-4073-8f2c-64b52d9f959d nodeName:}" failed. No retries permitted until 2026-07-23 17:57:32.68549965 +0000 UTC m=+2.126998228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret") pod "global-pull-secret-syncer-865dp" (UID: "1d669e26-a0e5-4073-8f2c-64b52d9f959d") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185524 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-run-openvswitch\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185532 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-var-lib-openvswitch\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185541 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-ovn-node-metrics-cert\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185553 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-env-overrides\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185576 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-run-openvswitch\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185605 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d669e26-a0e5-4073-8f2c-64b52d9f959d-kubelet-config\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-sys-fs\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185675 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdk4\" (UniqueName: \"kubernetes.io/projected/cccf0c11-35a3-4785-8723-5f3c8002e4b7-kube-api-access-jmdk4\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.186182 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185703 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d669e26-a0e5-4073-8f2c-64b52d9f959d-kubelet-config\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185713 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-sys-fs\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185708 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e2c7a4c9-5c1f-45df-a280-ed8a43b90a93-konnectivity-ca\") pod \"konnectivity-agent-9rmwf\" (UID: \"e2c7a4c9-5c1f-45df-a280-ed8a43b90a93\") " pod="kube-system/konnectivity-agent-9rmwf" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185758 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-host\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9frz\" (UniqueName: \"kubernetes.io/projected/9a34406b-fbaf-4512-871f-768a1a43ddc3-kube-api-access-t9frz\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-ovnkube-config\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x6w5\" (UniqueName: \"kubernetes.io/projected/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-kube-api-access-7x6w5\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185859 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-host\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185872 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d669e26-a0e5-4073-8f2c-64b52d9f959d-dbus\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-lib-modules\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.185985 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d669e26-a0e5-4073-8f2c-64b52d9f959d-dbus\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-tuned\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186053 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-lib-modules\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186055 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e2c7a4c9-5c1f-45df-a280-ed8a43b90a93-agent-certs\") pod \"konnectivity-agent-9rmwf\" (UID: \"e2c7a4c9-5c1f-45df-a280-ed8a43b90a93\") " pod="kube-system/konnectivity-agent-9rmwf" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-run-systemd\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186122 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-socket-dir\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-device-dir\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186162 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-run-systemd\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.187018 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186193 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-sysconfig\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186227 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-device-dir\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-kubernetes\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186257 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-socket-dir\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-kubelet\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-sysconfig\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-log-socket\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186306 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-kubelet\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-cni-netd\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186331 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e2c7a4c9-5c1f-45df-a280-ed8a43b90a93-konnectivity-ca\") pod \"konnectivity-agent-9rmwf\" (UID: \"e2c7a4c9-5c1f-45df-a280-ed8a43b90a93\") " pod="kube-system/konnectivity-agent-9rmwf" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-systemd\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186383 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-sys\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186402 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-cni-netd\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186418 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad-tmp-dir\") pod \"node-resolver-gwvmh\" (UID: \"d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad\") " pod="openshift-dns/node-resolver-gwvmh" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186445 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-sys\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186335 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-kubernetes\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186457 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-systemd\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186359 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-log-socket\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.187853 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xq4r\" (UniqueName: \"kubernetes.io/projected/6098df05-0cf3-46ae-83ea-f16c1edf1260-kube-api-access-8xq4r\") pod \"node-ca-x82j5\" (UID: \"6098df05-0cf3-46ae-83ea-f16c1edf1260\") " pod="openshift-image-registry/node-ca-x82j5" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjf7\" (UniqueName: \"kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7\") pod \"network-check-target-tcz9v\" (UID: \"d3acdd92-76f4-4cf9-b91e-4ba8016f441a\") " pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186582 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-slash\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-sysctl-d\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a34406b-fbaf-4512-871f-768a1a43ddc3-tmp\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4s7k\" (UniqueName: \"kubernetes.io/projected/d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad-kube-api-access-s4s7k\") pod \"node-resolver-gwvmh\" (UID: \"d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad\") " pod="openshift-dns/node-resolver-gwvmh" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-ovnkube-script-lib\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-run\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-cni-bin\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186761 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad-tmp-dir\") pod \"node-resolver-gwvmh\" (UID: \"d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad\") " pod="openshift-dns/node-resolver-gwvmh" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6098df05-0cf3-46ae-83ea-f16c1edf1260-serviceca\") pod \"node-ca-x82j5\" (UID: \"6098df05-0cf3-46ae-83ea-f16c1edf1260\") " pod="openshift-image-registry/node-ca-x82j5" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186821 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-registration-dir\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186861 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186856 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-var-lib-kubelet\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186891 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.188518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186909 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-var-lib-kubelet\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.189177 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186963 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-slash\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.189177 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.186983 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cccf0c11-35a3-4785-8723-5f3c8002e4b7-registration-dir\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.189177 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.187133 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-run\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.189177 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.187161 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-host-cni-bin\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.189177 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.187516 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-ovnkube-script-lib\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.189177 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.187632 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-sysctl-d\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.189177 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.187720 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6098df05-0cf3-46ae-83ea-f16c1edf1260-serviceca\") pod \"node-ca-x82j5\" (UID: \"6098df05-0cf3-46ae-83ea-f16c1edf1260\") " pod="openshift-image-registry/node-ca-x82j5" Jul 23 17:57:32.189177 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.187915 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-ovnkube-config\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.189177 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.188457 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-ovn-node-metrics-cert\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.189177 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.188528 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a34406b-fbaf-4512-871f-768a1a43ddc3-etc-tuned\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.189177 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.188691 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e2c7a4c9-5c1f-45df-a280-ed8a43b90a93-agent-certs\") pod \"konnectivity-agent-9rmwf\" (UID: \"e2c7a4c9-5c1f-45df-a280-ed8a43b90a93\") " pod="kube-system/konnectivity-agent-9rmwf" Jul 23 17:57:32.189177 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.189131 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a34406b-fbaf-4512-871f-768a1a43ddc3-tmp\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.206174 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.206147 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x6w5\" (UniqueName: \"kubernetes.io/projected/af7ec15e-8c6c-4094-85ca-37f2cc06aacd-kube-api-access-7x6w5\") pod \"ovnkube-node-rfsgl\" (UID: \"af7ec15e-8c6c-4094-85ca-37f2cc06aacd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.207294 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.207274 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9frz\" (UniqueName: \"kubernetes.io/projected/9a34406b-fbaf-4512-871f-768a1a43ddc3-kube-api-access-t9frz\") pod \"tuned-m24k7\" (UID: \"9a34406b-fbaf-4512-871f-768a1a43ddc3\") " pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.207525 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.207500 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdk4\" (UniqueName: \"kubernetes.io/projected/cccf0c11-35a3-4785-8723-5f3c8002e4b7-kube-api-access-jmdk4\") pod \"aws-ebs-csi-driver-node-5tq7b\" (UID: \"cccf0c11-35a3-4785-8723-5f3c8002e4b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.213108 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.213090 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:57:32.213108 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.213111 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:57:32.213229 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.213123 2564 projected.go:196] Error preparing data for projected volume kube-api-access-nhjf7 for pod openshift-network-diagnostics/network-check-target-tcz9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:57:32.213229 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.213179 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7 podName:d3acdd92-76f4-4cf9-b91e-4ba8016f441a nodeName:}" failed. No retries permitted until 2026-07-23 17:57:32.713165262 +0000 UTC m=+2.154663835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nhjf7" (UniqueName: "kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7") pod "network-check-target-tcz9v" (UID: "d3acdd92-76f4-4cf9-b91e-4ba8016f441a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:57:32.214011 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.213987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4s7k\" (UniqueName: \"kubernetes.io/projected/d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad-kube-api-access-s4s7k\") pod \"node-resolver-gwvmh\" (UID: \"d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad\") " pod="openshift-dns/node-resolver-gwvmh" Jul 23 17:57:32.214515 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.214497 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xq4r\" (UniqueName: \"kubernetes.io/projected/6098df05-0cf3-46ae-83ea-f16c1edf1260-kube-api-access-8xq4r\") pod \"node-ca-x82j5\" (UID: \"6098df05-0cf3-46ae-83ea-f16c1edf1260\") " pod="openshift-image-registry/node-ca-x82j5" Jul 23 17:57:32.314115 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.314072 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7zjxq" Jul 23 17:57:32.323751 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:32.323721 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded871313_07b6_4f58_aaed_314636ffcc3d.slice/crio-de17f43004b2eeff4c1c2d0aa21048ced5adbaa4e4711af12523ee07de79839f WatchSource:0}: Error finding container de17f43004b2eeff4c1c2d0aa21048ced5adbaa4e4711af12523ee07de79839f: Status 404 returned error can't find the container with id de17f43004b2eeff4c1c2d0aa21048ced5adbaa4e4711af12523ee07de79839f Jul 23 17:57:32.328428 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.328403 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-psq9w" Jul 23 17:57:32.335862 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:32.335824 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc385ce53_b210_4737_ae2b_a26aed9b4399.slice/crio-4845b328ec8a50196593bdb5c8fbddcbbc8be695b962ebccd30ad44cc37bf11d WatchSource:0}: Error finding container 4845b328ec8a50196593bdb5c8fbddcbbc8be695b962ebccd30ad44cc37bf11d: Status 404 returned error can't find the container with id 4845b328ec8a50196593bdb5c8fbddcbbc8be695b962ebccd30ad44cc37bf11d Jul 23 17:57:32.346027 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.346006 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8kbwh" Jul 23 17:57:32.352052 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.352034 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9rmwf" Jul 23 17:57:32.353493 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:32.353470 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04a87a4_9a0b_4840_8642_82500e37d420.slice/crio-3dd11ef200b67ede52973735d8e936c3ad1846557219eedeb47861e4264f3314 WatchSource:0}: Error finding container 3dd11ef200b67ede52973735d8e936c3ad1846557219eedeb47861e4264f3314: Status 404 returned error can't find the container with id 3dd11ef200b67ede52973735d8e936c3ad1846557219eedeb47861e4264f3314 Jul 23 17:57:32.359415 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:32.359390 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c7a4c9_5c1f_45df_a280_ed8a43b90a93.slice/crio-5b37d9e2ca8192272cb409279fcb434486ea1e805ec75ba164779e994ad4595d WatchSource:0}: Error finding container 5b37d9e2ca8192272cb409279fcb434486ea1e805ec75ba164779e994ad4595d: Status 404 returned error can't find the container with id 5b37d9e2ca8192272cb409279fcb434486ea1e805ec75ba164779e994ad4595d Jul 23 17:57:32.376732 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.376711 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gwvmh" Jul 23 17:57:32.382845 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.382825 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:32.384763 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:32.384728 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2dacd0d_9d63_4da0_ad08_1c36f4fa3dad.slice/crio-ae3481e040563faadaa41a515a4e00d66212a18304bae945f86131f3ac51ec68 WatchSource:0}: Error finding container ae3481e040563faadaa41a515a4e00d66212a18304bae945f86131f3ac51ec68: Status 404 returned error can't find the container with id ae3481e040563faadaa41a515a4e00d66212a18304bae945f86131f3ac51ec68 Jul 23 17:57:32.391243 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.391222 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" Jul 23 17:57:32.391485 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:32.391466 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf7ec15e_8c6c_4094_85ca_37f2cc06aacd.slice/crio-3e1d4a5e03948fd70ab556d65e0e8e4de8793253163794f1bbc54245cc8074b8 WatchSource:0}: Error finding container 3e1d4a5e03948fd70ab556d65e0e8e4de8793253163794f1bbc54245cc8074b8: Status 404 returned error can't find the container with id 3e1d4a5e03948fd70ab556d65e0e8e4de8793253163794f1bbc54245cc8074b8 Jul 23 17:57:32.398553 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:32.398527 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccf0c11_35a3_4785_8723_5f3c8002e4b7.slice/crio-8169a2632195f006440ccc59dff4e5ea6bd1f86a71d71e7cff67713acae11d50 WatchSource:0}: Error finding container 8169a2632195f006440ccc59dff4e5ea6bd1f86a71d71e7cff67713acae11d50: Status 404 returned error can't find the container with id 8169a2632195f006440ccc59dff4e5ea6bd1f86a71d71e7cff67713acae11d50 Jul 23 17:57:32.411037 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.411018 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m24k7" Jul 23 17:57:32.411349 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.411332 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:57:32.414759 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.414736 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x82j5" Jul 23 17:57:32.420027 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:32.420006 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a34406b_fbaf_4512_871f_768a1a43ddc3.slice/crio-8738fad9ec40414b04c3b251a3bb6505f382a116a76944ff6d4e78d80cd9d77a WatchSource:0}: Error finding container 8738fad9ec40414b04c3b251a3bb6505f382a116a76944ff6d4e78d80cd9d77a: Status 404 returned error can't find the container with id 8738fad9ec40414b04c3b251a3bb6505f382a116a76944ff6d4e78d80cd9d77a Jul 23 17:57:32.423428 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:57:32.423403 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6098df05_0cf3_46ae_83ea_f16c1edf1260.slice/crio-f19eacea5da2903c9c47939a5647ecc31af779d91a28e048c139485458198193 WatchSource:0}: Error finding container f19eacea5da2903c9c47939a5647ecc31af779d91a28e048c139485458198193: Status 404 returned error can't find the container with id f19eacea5da2903c9c47939a5647ecc31af779d91a28e048c139485458198193 Jul 23 17:57:32.489480 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.489399 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:57:32.589338 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.589311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs\") pod \"network-metrics-daemon-lg9sf\" (UID: \"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26\") " pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:32.589515 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.589457 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:57:32.589561 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.589515 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs podName:b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26 nodeName:}" failed. No retries permitted until 2026-07-23 17:57:33.589500759 +0000 UTC m=+3.030999333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs") pod "network-metrics-daemon-lg9sf" (UID: "b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:57:32.690427 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.690393 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:32.690622 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.690519 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:57:32.690622 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.690585 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret podName:1d669e26-a0e5-4073-8f2c-64b52d9f959d nodeName:}" failed. No retries permitted until 2026-07-23 17:57:33.690565691 +0000 UTC m=+3.132064267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret") pod "global-pull-secret-syncer-865dp" (UID: "1d669e26-a0e5-4073-8f2c-64b52d9f959d") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:57:32.791515 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:32.791422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjf7\" (UniqueName: \"kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7\") pod \"network-check-target-tcz9v\" (UID: \"d3acdd92-76f4-4cf9-b91e-4ba8016f441a\") " pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:32.791685 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.791632 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:57:32.791685 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.791678 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:57:32.791825 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.791694 2564 projected.go:196] Error preparing data for projected volume kube-api-access-nhjf7 for pod openshift-network-diagnostics/network-check-target-tcz9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:57:32.791825 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:32.791758 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7 podName:d3acdd92-76f4-4cf9-b91e-4ba8016f441a nodeName:}" failed. No retries permitted until 2026-07-23 17:57:33.791738596 +0000 UTC m=+3.233237188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhjf7" (UniqueName: "kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7") pod "network-check-target-tcz9v" (UID: "d3acdd92-76f4-4cf9-b91e-4ba8016f441a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:57:33.052126 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.052011 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 17:52:32 +0000 UTC" deadline="2028-01-20 14:38:10.757860637 +0000 UTC" Jul 23 17:57:33.052126 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.052077 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13100h40m37.705788025s" Jul 23 17:57:33.064710 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.064679 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:57:33.128320 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.128151 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" event={"ID":"cccf0c11-35a3-4785-8723-5f3c8002e4b7","Type":"ContainerStarted","Data":"8169a2632195f006440ccc59dff4e5ea6bd1f86a71d71e7cff67713acae11d50"} Jul 23 17:57:33.131519 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.131463 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" event={"ID":"af7ec15e-8c6c-4094-85ca-37f2cc06aacd","Type":"ContainerStarted","Data":"3e1d4a5e03948fd70ab556d65e0e8e4de8793253163794f1bbc54245cc8074b8"} Jul 23 17:57:33.141435 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.141402 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gwvmh" event={"ID":"d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad","Type":"ContainerStarted","Data":"ae3481e040563faadaa41a515a4e00d66212a18304bae945f86131f3ac51ec68"} Jul 23 17:57:33.150108 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.149993 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9rmwf" event={"ID":"e2c7a4c9-5c1f-45df-a280-ed8a43b90a93","Type":"ContainerStarted","Data":"5b37d9e2ca8192272cb409279fcb434486ea1e805ec75ba164779e994ad4595d"} Jul 23 17:57:33.162368 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.162332 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-psq9w" event={"ID":"c385ce53-b210-4737-ae2b-a26aed9b4399","Type":"ContainerStarted","Data":"4845b328ec8a50196593bdb5c8fbddcbbc8be695b962ebccd30ad44cc37bf11d"} Jul 23 17:57:33.178172 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.175564 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zjxq" event={"ID":"ed871313-07b6-4f58-aaed-314636ffcc3d","Type":"ContainerStarted","Data":"de17f43004b2eeff4c1c2d0aa21048ced5adbaa4e4711af12523ee07de79839f"} Jul 23 17:57:33.192116 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.192062 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-147.ec2.internal" event={"ID":"4a44435e6f3df5d50b47d277e77112d1","Type":"ContainerStarted","Data":"bfd22fbe9928f0078f3d0d89fc41f7f26d46a1987a5b656163131c8c9f76ee23"} Jul 23 17:57:33.198923 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.198873 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x82j5" event={"ID":"6098df05-0cf3-46ae-83ea-f16c1edf1260","Type":"ContainerStarted","Data":"f19eacea5da2903c9c47939a5647ecc31af779d91a28e048c139485458198193"} Jul 23 17:57:33.202789 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.202759 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m24k7" event={"ID":"9a34406b-fbaf-4512-871f-768a1a43ddc3","Type":"ContainerStarted","Data":"8738fad9ec40414b04c3b251a3bb6505f382a116a76944ff6d4e78d80cd9d77a"} Jul 23 17:57:33.223082 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.223048 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8kbwh" event={"ID":"d04a87a4-9a0b-4840-8642-82500e37d420","Type":"ContainerStarted","Data":"3dd11ef200b67ede52973735d8e936c3ad1846557219eedeb47861e4264f3314"} Jul 23 17:57:33.231499 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.231399 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal" event={"ID":"1a43e806b4b4d8f64fbb04742acee7cb","Type":"ContainerStarted","Data":"483c1de3cb33dc6974aaf61a853c618330eddaa75652f7d832d1c55b1c8aee20"} Jul 23 17:57:33.599217 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.599073 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs\") pod \"network-metrics-daemon-lg9sf\" (UID: \"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26\") " pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:33.599478 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:33.599310 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:57:33.599478 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:33.599381 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs podName:b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26 nodeName:}" failed. No retries permitted until 2026-07-23 17:57:35.599360633 +0000 UTC m=+5.040859207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs") pod "network-metrics-daemon-lg9sf" (UID: "b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:57:33.700037 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.699853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:33.700836 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:33.700366 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:57:33.700836 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:33.700465 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret podName:1d669e26-a0e5-4073-8f2c-64b52d9f959d nodeName:}" failed. No retries permitted until 2026-07-23 17:57:35.700431127 +0000 UTC m=+5.141929703 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret") pod "global-pull-secret-syncer-865dp" (UID: "1d669e26-a0e5-4073-8f2c-64b52d9f959d") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:57:33.800928 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:33.800823 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjf7\" (UniqueName: \"kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7\") pod \"network-check-target-tcz9v\" (UID: \"d3acdd92-76f4-4cf9-b91e-4ba8016f441a\") " pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:33.801126 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:33.801029 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:57:33.801126 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:33.801055 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:57:33.801126 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:33.801068 2564 projected.go:196] Error preparing data for projected volume kube-api-access-nhjf7 for pod openshift-network-diagnostics/network-check-target-tcz9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:57:33.801289 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:33.801129 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7 podName:d3acdd92-76f4-4cf9-b91e-4ba8016f441a nodeName:}" failed. No retries permitted until 2026-07-23 17:57:35.801109958 +0000 UTC m=+5.242608535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhjf7" (UniqueName: "kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7") pod "network-check-target-tcz9v" (UID: "d3acdd92-76f4-4cf9-b91e-4ba8016f441a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:57:34.053180 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:34.053054 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 17:52:32 +0000 UTC" deadline="2028-05-07 18:50:23.543605842 +0000 UTC" Jul 23 17:57:34.053180 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:34.053125 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15696h52m49.49048483s" Jul 23 17:57:34.112949 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:34.111879 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:34.112949 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:34.112212 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:57:34.112949 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:34.112317 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:34.112949 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:34.112405 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:57:34.112949 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:34.112472 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:34.112949 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:34.112563 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:57:35.617764 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:35.617726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs\") pod \"network-metrics-daemon-lg9sf\" (UID: \"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26\") " pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:35.618250 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:35.617947 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:57:35.618250 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:35.618044 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs podName:b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26 nodeName:}" failed. No retries permitted until 2026-07-23 17:57:39.618022996 +0000 UTC m=+9.059521570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs") pod "network-metrics-daemon-lg9sf" (UID: "b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:57:35.719219 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:35.719176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:35.719415 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:35.719394 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:57:35.719555 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:35.719538 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret podName:1d669e26-a0e5-4073-8f2c-64b52d9f959d nodeName:}" failed. No retries permitted until 2026-07-23 17:57:39.719513873 +0000 UTC m=+9.161012452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret") pod "global-pull-secret-syncer-865dp" (UID: "1d669e26-a0e5-4073-8f2c-64b52d9f959d") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:57:35.821430 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:35.820609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjf7\" (UniqueName: \"kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7\") pod \"network-check-target-tcz9v\" (UID: \"d3acdd92-76f4-4cf9-b91e-4ba8016f441a\") " pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:35.821430 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:35.820826 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:57:35.821430 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:35.820863 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:57:35.821430 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:35.820878 2564 projected.go:196] Error preparing data for projected volume kube-api-access-nhjf7 for pod openshift-network-diagnostics/network-check-target-tcz9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:57:35.821430 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:35.820957 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7 podName:d3acdd92-76f4-4cf9-b91e-4ba8016f441a nodeName:}" failed. No retries permitted until 2026-07-23 17:57:39.8209335 +0000 UTC m=+9.262432098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhjf7" (UniqueName: "kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7") pod "network-check-target-tcz9v" (UID: "d3acdd92-76f4-4cf9-b91e-4ba8016f441a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:57:36.112241 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:36.112147 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:36.112425 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:36.112302 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:57:36.112703 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:36.112680 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:36.112810 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:36.112788 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:57:36.112911 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:36.112844 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:36.112980 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:36.112940 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:57:38.112199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:38.112007 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:38.112199 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:38.112149 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:57:38.112791 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:38.112379 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:38.112791 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:38.112489 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:57:38.112791 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:38.112561 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:38.113135 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:38.113093 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:57:39.655755 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:39.655709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs\") pod \"network-metrics-daemon-lg9sf\" (UID: \"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26\") " pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:39.656274 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:39.655925 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:57:39.656274 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:39.656002 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs podName:b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26 nodeName:}" failed. No retries permitted until 2026-07-23 17:57:47.655981131 +0000 UTC m=+17.097479705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs") pod "network-metrics-daemon-lg9sf" (UID: "b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:57:39.756202 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:39.756163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:39.756377 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:39.756343 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:57:39.756456 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:39.756430 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret podName:1d669e26-a0e5-4073-8f2c-64b52d9f959d nodeName:}" failed. No retries permitted until 2026-07-23 17:57:47.756405445 +0000 UTC m=+17.197904033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret") pod "global-pull-secret-syncer-865dp" (UID: "1d669e26-a0e5-4073-8f2c-64b52d9f959d") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:57:39.857117 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:39.857025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjf7\" (UniqueName: \"kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7\") pod \"network-check-target-tcz9v\" (UID: \"d3acdd92-76f4-4cf9-b91e-4ba8016f441a\") " pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:39.857301 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:39.857201 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:57:39.857301 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:39.857228 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:57:39.857301 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:39.857239 2564 projected.go:196] Error preparing data for projected volume kube-api-access-nhjf7 for pod openshift-network-diagnostics/network-check-target-tcz9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:57:39.857301 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:39.857298 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7 podName:d3acdd92-76f4-4cf9-b91e-4ba8016f441a nodeName:}" failed. No retries permitted until 2026-07-23 17:57:47.85727665 +0000 UTC m=+17.298775224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhjf7" (UniqueName: "kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7") pod "network-check-target-tcz9v" (UID: "d3acdd92-76f4-4cf9-b91e-4ba8016f441a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:57:40.111592 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:40.111278 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:40.111592 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:40.111422 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:40.111592 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:40.111422 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:40.111592 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:40.111417 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:57:40.111592 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:40.111556 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:57:40.111939 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:40.111673 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:57:42.111893 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:42.111852 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:42.112522 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:42.111897 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:42.112522 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:42.111969 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:57:42.112522 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:42.112022 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:57:42.112522 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:42.112080 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:42.112522 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:42.112173 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:57:44.111650 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:44.111615 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:44.112092 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:44.111743 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:44.112092 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:44.111748 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:44.112092 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:44.111745 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:57:44.112092 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:44.111876 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:57:44.112092 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:44.111974 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:57:46.112015 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:46.111975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:46.112015 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:46.111996 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:46.112496 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:46.111993 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:46.112496 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:46.112116 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:57:46.112496 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:46.112256 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:57:46.112496 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:46.112351 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:57:47.715622 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:47.715581 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs\") pod \"network-metrics-daemon-lg9sf\" (UID: \"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26\") " pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:47.716113 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:47.715704 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:57:47.716113 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:47.715772 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs podName:b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:03.715753019 +0000 UTC m=+33.157251604 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs") pod "network-metrics-daemon-lg9sf" (UID: "b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:57:47.816551 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:47.816513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:47.816773 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:47.816632 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:57:47.816773 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:47.816690 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret podName:1d669e26-a0e5-4073-8f2c-64b52d9f959d nodeName:}" failed. No retries permitted until 2026-07-23 17:58:03.816672092 +0000 UTC m=+33.258170666 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret") pod "global-pull-secret-syncer-865dp" (UID: "1d669e26-a0e5-4073-8f2c-64b52d9f959d") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:57:47.917364 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:47.917323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjf7\" (UniqueName: \"kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7\") pod \"network-check-target-tcz9v\" (UID: \"d3acdd92-76f4-4cf9-b91e-4ba8016f441a\") " pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:47.917547 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:47.917518 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:57:47.917547 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:47.917545 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:57:47.917658 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:47.917559 2564 projected.go:196] Error preparing data for projected volume kube-api-access-nhjf7 for pod openshift-network-diagnostics/network-check-target-tcz9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:57:47.917658 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:47.917627 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7 podName:d3acdd92-76f4-4cf9-b91e-4ba8016f441a nodeName:}" failed. No retries permitted until 2026-07-23 17:58:03.917605655 +0000 UTC m=+33.359104242 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhjf7" (UniqueName: "kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7") pod "network-check-target-tcz9v" (UID: "d3acdd92-76f4-4cf9-b91e-4ba8016f441a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:57:48.112183 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:48.112101 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:48.112348 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:48.112101 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:48.112348 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:48.112238 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:57:48.112348 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:48.112101 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:48.112348 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:48.112297 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:57:48.112564 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:48.112396 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:57:50.111821 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:50.111769 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:50.112350 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:50.111834 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:50.112350 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:50.111772 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:50.112350 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:50.111923 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:57:50.112350 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:50.111998 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:57:50.112350 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:50.112069 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:57:52.111741 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:52.111684 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:52.111741 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:52.111728 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:52.112274 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:52.111732 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:52.112274 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:52.111875 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:57:52.112274 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:52.111961 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:57:52.112274 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:52.112072 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:57:54.111519 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.111225 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:54.112147 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.111239 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:54.112147 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:54.111593 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:57:54.112147 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:54.111689 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:57:54.112147 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.111226 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:54.112147 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:54.111777 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:57:54.285237 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.285156 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" event={"ID":"cccf0c11-35a3-4785-8723-5f3c8002e4b7","Type":"ContainerStarted","Data":"3463a906b0561f8cbfff210a0969e3ae196a91a92368ca60ff69ecb45061bed0"} Jul 23 17:57:54.288559 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.288526 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" event={"ID":"af7ec15e-8c6c-4094-85ca-37f2cc06aacd","Type":"ContainerStarted","Data":"b169ee7fcaf5a3cf4ae0103a0542c41ae6080b3e9aaa329e98c087a008022541"} Jul 23 17:57:54.288699 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.288611 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" event={"ID":"af7ec15e-8c6c-4094-85ca-37f2cc06aacd","Type":"ContainerStarted","Data":"4e56a95b608886cf60dc0f67f2f1bcb93923b40eca3c96ecc88e099b51575a40"} Jul 23 17:57:54.288699 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.288659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" event={"ID":"af7ec15e-8c6c-4094-85ca-37f2cc06aacd","Type":"ContainerStarted","Data":"ea47b1bb493dc2a1a38e09b3bd35383f3a2f56ead696059ef8c2a283c3189c16"} Jul 23 17:57:54.288830 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.288702 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" event={"ID":"af7ec15e-8c6c-4094-85ca-37f2cc06aacd","Type":"ContainerStarted","Data":"bde4d32d990dd119a4eb6f82037496c158799a582927cba8cd7fb87fdd874499"} Jul 23 17:57:54.288830 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.288749 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" event={"ID":"af7ec15e-8c6c-4094-85ca-37f2cc06aacd","Type":"ContainerStarted","Data":"6ace1216a74c0e288174c4ab6f1fd770c0bb25380b52149a495db062af178905"} Jul 23 17:57:54.288830 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.288798 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" event={"ID":"af7ec15e-8c6c-4094-85ca-37f2cc06aacd","Type":"ContainerStarted","Data":"c70ce78ff9d7975abf0bd73a4f8ae2d920767d8981d688cfaa45f8b958797045"} Jul 23 17:57:54.290108 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.290068 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gwvmh" event={"ID":"d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad","Type":"ContainerStarted","Data":"ecc8c3243b0b7c1c1192a080d59cdf7e8d48eb453d5a2d809b58c32579dc4fb0"} Jul 23 17:57:54.291668 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.291642 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9rmwf" event={"ID":"e2c7a4c9-5c1f-45df-a280-ed8a43b90a93","Type":"ContainerStarted","Data":"cecfd4fca55d3720f4a2daa2ee1a27d9262c58ec8e653f1b5fda91ad3b73e2b9"} Jul 23 17:57:54.293189 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.293162 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-psq9w" event={"ID":"c385ce53-b210-4737-ae2b-a26aed9b4399","Type":"ContainerStarted","Data":"eec4b219c21c61eae9c7d9559b07362216b21ae584224922206586eda62ceb71"} Jul 23 17:57:54.295058 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.295024 2564 generic.go:356] "Generic (PLEG): container finished" podID="ed871313-07b6-4f58-aaed-314636ffcc3d" containerID="a3e6963a40281a37dbf198490d0d5edc078242554bcbec26dfd8d867d51ecdf8" exitCode=0 Jul 23 17:57:54.295154 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.295083 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zjxq" event={"ID":"ed871313-07b6-4f58-aaed-314636ffcc3d","Type":"ContainerDied","Data":"a3e6963a40281a37dbf198490d0d5edc078242554bcbec26dfd8d867d51ecdf8"} Jul 23 17:57:54.296776 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.296747 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-147.ec2.internal" event={"ID":"4a44435e6f3df5d50b47d277e77112d1","Type":"ContainerStarted","Data":"cf9370e01827b30cddb59a08f2be506c08fc915c844400a6c1982e740b5693fc"} Jul 23 17:57:54.298449 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.298423 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x82j5" event={"ID":"6098df05-0cf3-46ae-83ea-f16c1edf1260","Type":"ContainerStarted","Data":"00496f502c28c81dbddc656b29228bb83e4ba0cfd786ad5f3a0adcc60f982511"} Jul 23 17:57:54.300163 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.300135 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m24k7" event={"ID":"9a34406b-fbaf-4512-871f-768a1a43ddc3","Type":"ContainerStarted","Data":"512a780ab87816fe9ab95c493f0cd5f592942ef6623a4f4182418acb69441226"} Jul 23 17:57:54.301829 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.301790 2564 generic.go:356] "Generic (PLEG): container finished" podID="1a43e806b4b4d8f64fbb04742acee7cb" containerID="feaa8d33fac471f962fbb6420ea4296fa504f7ff1068bf17a9a36dac551f871a" exitCode=0 Jul 23 17:57:54.301922 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.301840 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal" event={"ID":"1a43e806b4b4d8f64fbb04742acee7cb","Type":"ContainerDied","Data":"feaa8d33fac471f962fbb6420ea4296fa504f7ff1068bf17a9a36dac551f871a"} Jul 23 17:57:54.309467 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.309427 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gwvmh" podStartSLOduration=2.611501961 podStartE2EDuration="23.309409771s" totalImagesPullingTime="20.69790781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:57:54.30893978 +0000 UTC m=+23.750438376" watchObservedRunningTime="2026-07-23 17:57:54.309409771 +0000 UTC m=+23.750908368" Jul 23 17:57:54.347405 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.347351 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9rmwf" podStartSLOduration=11.029914844 podStartE2EDuration="23.347328721s" totalImagesPullingTime="12.317413877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:57:54.326745867 +0000 UTC m=+23.768244461" watchObservedRunningTime="2026-07-23 17:57:54.347328721 +0000 UTC m=+23.788827315" Jul 23 17:57:54.365145 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.365083 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m24k7" podStartSLOduration=2.698769364 podStartE2EDuration="23.36506325s" totalImagesPullingTime="20.666293886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:57:54.347324585 +0000 UTC m=+23.788823179" watchObservedRunningTime="2026-07-23 17:57:54.36506325 +0000 UTC m=+23.806561845" Jul 23 17:57:54.365317 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.365212 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x82j5" podStartSLOduration=1.704259416 podStartE2EDuration="22.365205009s" totalImagesPullingTime="20.660945593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:57:54.365016887 +0000 UTC m=+23.806515482" watchObservedRunningTime="2026-07-23 17:57:54.365205009 +0000 UTC m=+23.806703604" Jul 23 17:57:54.427998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.427944 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-147.ec2.internal" podStartSLOduration=22.427922071 podStartE2EDuration="22.427922071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:57:54.404857037 +0000 UTC m=+23.846355631" watchObservedRunningTime="2026-07-23 17:57:54.427922071 +0000 UTC m=+23.869420665" Jul 23 17:57:54.449674 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.449629 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-psq9w" podStartSLOduration=2.6903263429999997 podStartE2EDuration="23.449613245s" totalImagesPullingTime="20.759286902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:57:54.449600088 +0000 UTC m=+23.891098767" watchObservedRunningTime="2026-07-23 17:57:54.449613245 +0000 UTC m=+23.891111839" Jul 23 17:57:54.868572 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.868322 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9rmwf" Jul 23 17:57:54.868884 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:54.868869 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9rmwf" Jul 23 17:57:55.048364 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:55.048337 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 17:57:55.306450 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:55.306365 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8kbwh" event={"ID":"d04a87a4-9a0b-4840-8642-82500e37d420","Type":"ContainerStarted","Data":"9cd9868c45092ec5364cd36fe07c0819e64f4fb8d73f9726a3fad78a66c8e574"} Jul 23 17:57:55.308618 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:55.308592 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal" event={"ID":"1a43e806b4b4d8f64fbb04742acee7cb","Type":"ContainerStarted","Data":"32661fb7cb2393244665d29c93bf7ecdd5aa0e43bc7af20f8cb74aca8e6bec9f"} Jul 23 17:57:55.310719 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:55.310663 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" event={"ID":"cccf0c11-35a3-4785-8723-5f3c8002e4b7","Type":"ContainerStarted","Data":"7ffdda4d5b65a74f75a9c5e227c0ffcb04d3ba76700714374a13ef43cb7926d0"} Jul 23 17:57:55.332216 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:55.332164 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8kbwh" podStartSLOduration=3.602291948 podStartE2EDuration="24.332141828s" totalImagesPullingTime="20.72984988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:57:55.332124972 +0000 UTC m=+24.773623567" watchObservedRunningTime="2026-07-23 17:57:55.332141828 +0000 UTC m=+24.773640422" Jul 23 17:57:55.355419 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:55.355340 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-147.ec2.internal" podStartSLOduration=23.355325591 podStartE2EDuration="23.355325591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:57:55.354299935 +0000 UTC m=+24.795798528" watchObservedRunningTime="2026-07-23 17:57:55.355325591 +0000 UTC m=+24.796824185" Jul 23 17:57:56.036585 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:56.036483 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T17:57:55.048358711Z","UUID":"0db11dd9-e963-4c18-88d2-20a9d0f90fdc","Handler":null,"Name":"","Endpoint":""} Jul 23 17:57:56.038141 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:56.038117 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 17:57:56.038263 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:56.038174 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 17:57:56.111912 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:56.111868 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:56.112091 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:56.111869 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:56.112091 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:56.111869 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:56.112171 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:56.112108 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:57:56.112207 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:56.112178 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:57:56.112207 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:56.111992 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:57:56.312797 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:56.312691 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 23 17:57:57.317506 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:57.317463 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" event={"ID":"cccf0c11-35a3-4785-8723-5f3c8002e4b7","Type":"ContainerStarted","Data":"efa5d7c60d7bbd6fbc332c9778bea5935d174490448c5912ffc7a1da8fd5e076"} Jul 23 17:57:57.321660 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:57.321627 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" event={"ID":"af7ec15e-8c6c-4094-85ca-37f2cc06aacd","Type":"ContainerStarted","Data":"0a6d6b6ca7951ef623ced52516357cf9c0f9da9d0b29b3d8357728c45fef7459"} Jul 23 17:57:57.343695 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:57.343644 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tq7b" podStartSLOduration=3.209036181 podStartE2EDuration="26.343629527s" totalImagesPullingTime="23.134593346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:57:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:57:57.343023598 +0000 UTC m=+26.784522193" watchObservedRunningTime="2026-07-23 17:57:57.343629527 +0000 UTC m=+26.785128121" Jul 23 17:57:58.111423 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:58.111382 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:57:58.111608 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:58.111383 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:57:58.111608 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:58.111517 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:57:58.111608 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:58.111383 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:57:58.111777 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:58.111604 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:57:58.111777 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:57:58.111694 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:57:59.328518 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:59.328285 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" event={"ID":"af7ec15e-8c6c-4094-85ca-37f2cc06aacd","Type":"ContainerStarted","Data":"399e988e0606bcfe7b3181d594192c14aa2193cf9d5534354762e29f7e9fd30f"} Jul 23 17:57:59.329055 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:59.328576 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:59.330121 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:59.330096 2564 generic.go:356] "Generic (PLEG): container finished" podID="ed871313-07b6-4f58-aaed-314636ffcc3d" containerID="110f87062e781569741d59bc94b28db8d335c9ea5aadcb0308ed87c9704baefb" exitCode=0 Jul 23 17:57:59.330210 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:59.330142 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zjxq" event={"ID":"ed871313-07b6-4f58-aaed-314636ffcc3d","Type":"ContainerDied","Data":"110f87062e781569741d59bc94b28db8d335c9ea5aadcb0308ed87c9704baefb"} Jul 23 17:57:59.343775 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:59.343746 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:57:59.359759 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:57:59.359706 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" podStartSLOduration=7.432188859 podStartE2EDuration="28.359693287s" totalImagesPullingTime="20.927504428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:57:59.359549308 +0000 UTC m=+28.801047902" watchObservedRunningTime="2026-07-23 17:57:59.359693287 +0000 UTC m=+28.801191881" Jul 23 17:58:00.111367 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.111333 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:58:00.111579 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.111436 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:58:00.111579 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:00.111539 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:58:00.111579 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:00.111441 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:58:00.111579 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.111443 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:58:00.111736 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:00.111619 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:58:00.335593 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.335552 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:58:00.336432 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.335685 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:58:00.349911 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.349890 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:58:00.464141 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.464113 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9rmwf" Jul 23 17:58:00.464252 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.464238 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 23 17:58:00.464758 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.464737 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9rmwf" Jul 23 17:58:00.858901 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.858870 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tcz9v"] Jul 23 17:58:00.859051 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.858966 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:58:00.859051 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:00.859038 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:58:00.865861 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.865839 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-865dp"] Jul 23 17:58:00.865976 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.865928 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:58:00.866021 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:00.865999 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:58:00.871714 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.871687 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lg9sf"] Jul 23 17:58:00.871854 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:00.871786 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:58:00.871922 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:00.871890 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:58:01.339911 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:01.339877 2564 generic.go:356] "Generic (PLEG): container finished" podID="ed871313-07b6-4f58-aaed-314636ffcc3d" containerID="158f0ed9ecacdbb4ea6a11c6f2efc6d31dcae4d2d09555c7960a3eccac9a7c4d" exitCode=0 Jul 23 17:58:01.340456 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:01.339961 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zjxq" event={"ID":"ed871313-07b6-4f58-aaed-314636ffcc3d","Type":"ContainerDied","Data":"158f0ed9ecacdbb4ea6a11c6f2efc6d31dcae4d2d09555c7960a3eccac9a7c4d"} Jul 23 17:58:02.112099 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:02.112060 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:58:02.112306 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:02.112178 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:58:03.111276 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:03.111246 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:58:03.111693 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:03.111252 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:58:03.111693 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:03.111358 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:58:03.111693 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:03.111426 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:58:03.346346 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:03.346265 2564 generic.go:356] "Generic (PLEG): container finished" podID="ed871313-07b6-4f58-aaed-314636ffcc3d" containerID="f850e8fb5f87c0e8ea7a0c59e77ae3762a49d90f515408cf0775f00ce8a91826" exitCode=0 Jul 23 17:58:03.346346 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:03.346322 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zjxq" event={"ID":"ed871313-07b6-4f58-aaed-314636ffcc3d","Type":"ContainerDied","Data":"f850e8fb5f87c0e8ea7a0c59e77ae3762a49d90f515408cf0775f00ce8a91826"} Jul 23 17:58:03.735277 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:03.735192 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs\") pod \"network-metrics-daemon-lg9sf\" (UID: \"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26\") " pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:58:03.735438 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:03.735304 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:58:03.735438 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:03.735371 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs podName:b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:35.73535706 +0000 UTC m=+65.176855632 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs") pod "network-metrics-daemon-lg9sf" (UID: "b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:58:03.836495 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:03.836458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:58:03.836649 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:03.836608 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:58:03.836690 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:03.836674 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret podName:1d669e26-a0e5-4073-8f2c-64b52d9f959d nodeName:}" failed. No retries permitted until 2026-07-23 17:58:35.836657225 +0000 UTC m=+65.278155800 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret") pod "global-pull-secret-syncer-865dp" (UID: "1d669e26-a0e5-4073-8f2c-64b52d9f959d") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:58:03.937696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:03.937656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjf7\" (UniqueName: \"kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7\") pod \"network-check-target-tcz9v\" (UID: \"d3acdd92-76f4-4cf9-b91e-4ba8016f441a\") " pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:58:03.937895 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:03.937857 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:58:03.937895 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:03.937882 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:58:03.937895 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:03.937894 2564 projected.go:196] Error preparing data for projected volume kube-api-access-nhjf7 for pod openshift-network-diagnostics/network-check-target-tcz9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:58:03.938051 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:03.937960 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7 podName:d3acdd92-76f4-4cf9-b91e-4ba8016f441a nodeName:}" failed. No retries permitted until 2026-07-23 17:58:35.937941084 +0000 UTC m=+65.379439667 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhjf7" (UniqueName: "kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7") pod "network-check-target-tcz9v" (UID: "d3acdd92-76f4-4cf9-b91e-4ba8016f441a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:58:04.111455 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:04.111376 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:58:04.111908 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:04.111508 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:58:05.112015 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:05.111978 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:58:05.112408 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:05.112128 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-865dp" podUID="1d669e26-a0e5-4073-8f2c-64b52d9f959d" Jul 23 17:58:05.112408 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:05.112188 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:58:05.112408 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:05.112299 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcz9v" podUID="d3acdd92-76f4-4cf9-b91e-4ba8016f441a" Jul 23 17:58:06.111771 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.111724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:58:06.111966 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:06.111877 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg9sf" podUID="b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26" Jul 23 17:58:06.847798 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.847762 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-147.ec2.internal" event="NodeReady" Jul 23 17:58:06.848213 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.847942 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 17:58:06.900461 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.900419 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5nhzx"] Jul 23 17:58:06.903385 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.903360 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:06.903543 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.903510 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hvkrn"] Jul 23 17:58:06.905166 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.905148 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hvkrn" Jul 23 17:58:06.910030 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.909970 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 17:58:06.910030 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.910013 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 17:58:06.910393 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.910371 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 17:58:06.910491 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.910410 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 17:58:06.910491 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.910430 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-djwnl\"" Jul 23 17:58:06.910491 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.910464 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 17:58:06.910969 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.910951 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xlctx\"" Jul 23 17:58:06.923023 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.922962 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hvkrn"] Jul 23 17:58:06.923847 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:06.923785 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5nhzx"] Jul 23 17:58:07.062854 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.062761 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:07.062854 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.062821 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmkvv\" (UniqueName: \"kubernetes.io/projected/ce29f6ec-8310-4439-96e9-adaec2684196-kube-api-access-mmkvv\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:07.063067 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.062859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct6xp\" (UniqueName: \"kubernetes.io/projected/dee23d54-ccac-4849-8abd-3f25a806ca2e-kube-api-access-ct6xp\") pod \"ingress-canary-hvkrn\" (UID: \"dee23d54-ccac-4849-8abd-3f25a806ca2e\") " pod="openshift-ingress-canary/ingress-canary-hvkrn" Jul 23 17:58:07.063067 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.062885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce29f6ec-8310-4439-96e9-adaec2684196-tmp-dir\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:07.063067 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.062911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dee23d54-ccac-4849-8abd-3f25a806ca2e-cert\") pod \"ingress-canary-hvkrn\" (UID: \"dee23d54-ccac-4849-8abd-3f25a806ca2e\") " pod="openshift-ingress-canary/ingress-canary-hvkrn" Jul 23 17:58:07.063067 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.062969 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce29f6ec-8310-4439-96e9-adaec2684196-config-volume\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:07.112204 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.112161 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:58:07.112369 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.112163 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:58:07.116041 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.116013 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 17:58:07.116217 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.116139 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qd4dg\"" Jul 23 17:58:07.116217 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.116158 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 17:58:07.116217 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.116158 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 17:58:07.164135 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.164105 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:07.164298 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.164144 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmkvv\" (UniqueName: \"kubernetes.io/projected/ce29f6ec-8310-4439-96e9-adaec2684196-kube-api-access-mmkvv\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:07.164298 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.164171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct6xp\" (UniqueName: \"kubernetes.io/projected/dee23d54-ccac-4849-8abd-3f25a806ca2e-kube-api-access-ct6xp\") pod \"ingress-canary-hvkrn\" (UID: \"dee23d54-ccac-4849-8abd-3f25a806ca2e\") " pod="openshift-ingress-canary/ingress-canary-hvkrn" Jul 23 17:58:07.164298 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.164198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce29f6ec-8310-4439-96e9-adaec2684196-tmp-dir\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:07.164298 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:07.164272 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:58:07.164504 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:07.164333 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls podName:ce29f6ec-8310-4439-96e9-adaec2684196 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:07.664316201 +0000 UTC m=+37.105814776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls") pod "dns-default-5nhzx" (UID: "ce29f6ec-8310-4439-96e9-adaec2684196") : secret "dns-default-metrics-tls" not found Jul 23 17:58:07.164504 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.164382 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dee23d54-ccac-4849-8abd-3f25a806ca2e-cert\") pod \"ingress-canary-hvkrn\" (UID: \"dee23d54-ccac-4849-8abd-3f25a806ca2e\") " pod="openshift-ingress-canary/ingress-canary-hvkrn" Jul 23 17:58:07.164504 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.164452 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce29f6ec-8310-4439-96e9-adaec2684196-config-volume\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:07.164504 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.164470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce29f6ec-8310-4439-96e9-adaec2684196-tmp-dir\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:07.165020 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.164998 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce29f6ec-8310-4439-96e9-adaec2684196-config-volume\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:07.169075 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.169051 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dee23d54-ccac-4849-8abd-3f25a806ca2e-cert\") pod \"ingress-canary-hvkrn\" (UID: \"dee23d54-ccac-4849-8abd-3f25a806ca2e\") " pod="openshift-ingress-canary/ingress-canary-hvkrn" Jul 23 17:58:07.187227 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.187199 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmkvv\" (UniqueName: \"kubernetes.io/projected/ce29f6ec-8310-4439-96e9-adaec2684196-kube-api-access-mmkvv\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:07.187457 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.187436 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct6xp\" (UniqueName: \"kubernetes.io/projected/dee23d54-ccac-4849-8abd-3f25a806ca2e-kube-api-access-ct6xp\") pod \"ingress-canary-hvkrn\" (UID: \"dee23d54-ccac-4849-8abd-3f25a806ca2e\") " pod="openshift-ingress-canary/ingress-canary-hvkrn" Jul 23 17:58:07.220895 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.220862 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hvkrn" Jul 23 17:58:07.490704 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.490443 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hvkrn"] Jul 23 17:58:07.492985 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:58:07.492952 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee23d54_ccac_4849_8abd_3f25a806ca2e.slice/crio-22926dadf3057f72b5c32ff260dc71d9b6c63a5e87d9fbf48244175737208f1e WatchSource:0}: Error finding container 22926dadf3057f72b5c32ff260dc71d9b6c63a5e87d9fbf48244175737208f1e: Status 404 returned error can't find the container with id 22926dadf3057f72b5c32ff260dc71d9b6c63a5e87d9fbf48244175737208f1e Jul 23 17:58:07.669040 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:07.668947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:07.669191 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:07.669106 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:58:07.669191 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:07.669169 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls podName:ce29f6ec-8310-4439-96e9-adaec2684196 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:08.669151785 +0000 UTC m=+38.110650365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls") pod "dns-default-5nhzx" (UID: "ce29f6ec-8310-4439-96e9-adaec2684196") : secret "dns-default-metrics-tls" not found Jul 23 17:58:08.112021 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:08.111940 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:58:08.115145 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:08.115113 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 17:58:08.115145 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:08.115124 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7wdhp\"" Jul 23 17:58:08.362875 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:08.362776 2564 generic.go:356] "Generic (PLEG): container finished" podID="ed871313-07b6-4f58-aaed-314636ffcc3d" containerID="c61b6ef9e0d7a7e4dc1a64ae8d15cf449a16caaf0dbcaa8f033284bb700df500" exitCode=0 Jul 23 17:58:08.363017 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:08.362891 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zjxq" event={"ID":"ed871313-07b6-4f58-aaed-314636ffcc3d","Type":"ContainerDied","Data":"c61b6ef9e0d7a7e4dc1a64ae8d15cf449a16caaf0dbcaa8f033284bb700df500"} Jul 23 17:58:08.364575 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:08.364545 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hvkrn" event={"ID":"dee23d54-ccac-4849-8abd-3f25a806ca2e","Type":"ContainerStarted","Data":"22926dadf3057f72b5c32ff260dc71d9b6c63a5e87d9fbf48244175737208f1e"} Jul 23 17:58:08.675923 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:08.675839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:08.676069 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:08.675994 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:58:08.676069 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:08.676057 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls podName:ce29f6ec-8310-4439-96e9-adaec2684196 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:10.676039039 +0000 UTC m=+40.117537625 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls") pod "dns-default-5nhzx" (UID: "ce29f6ec-8310-4439-96e9-adaec2684196") : secret "dns-default-metrics-tls" not found Jul 23 17:58:09.373860 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:09.373821 2564 generic.go:356] "Generic (PLEG): container finished" podID="ed871313-07b6-4f58-aaed-314636ffcc3d" containerID="f1a367c9dc366aa7a5c37064b06cb784bd778ac3dc00358d6997bcaf00c7a33d" exitCode=0 Jul 23 17:58:09.374425 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:09.373885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zjxq" event={"ID":"ed871313-07b6-4f58-aaed-314636ffcc3d","Type":"ContainerDied","Data":"f1a367c9dc366aa7a5c37064b06cb784bd778ac3dc00358d6997bcaf00c7a33d"} Jul 23 17:58:10.378356 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:10.378149 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hvkrn" event={"ID":"dee23d54-ccac-4849-8abd-3f25a806ca2e","Type":"ContainerStarted","Data":"c11356d3329b59882b105f3dd6a3c7f4c83b5a95116c7fcb9dc8c79234301737"} Jul 23 17:58:10.381415 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:10.381392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zjxq" event={"ID":"ed871313-07b6-4f58-aaed-314636ffcc3d","Type":"ContainerStarted","Data":"0493283648c8629022cc0e1000539f22cbf6af4fc8747485b42bfcefd0e37986"} Jul 23 17:58:10.396854 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:10.396788 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hvkrn" podStartSLOduration=2.062921205 podStartE2EDuration="4.396773262s" totalImagesPullingTime="2.333852057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:58:10.3966906 +0000 UTC m=+39.838189217" watchObservedRunningTime="2026-07-23 17:58:10.396773262 +0000 UTC m=+39.838271855" Jul 23 17:58:10.424219 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:10.424168 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7zjxq" podStartSLOduration=8.204892971 podStartE2EDuration="39.424154608s" totalImagesPullingTime="31.219261637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:57:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:58:10.423960874 +0000 UTC m=+39.865459469" watchObservedRunningTime="2026-07-23 17:58:10.424154608 +0000 UTC m=+39.865653212" Jul 23 17:58:10.690063 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:10.689970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:10.690213 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:10.690097 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:58:10.690213 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:10.690147 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls podName:ce29f6ec-8310-4439-96e9-adaec2684196 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:14.690135273 +0000 UTC m=+44.131633847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls") pod "dns-default-5nhzx" (UID: "ce29f6ec-8310-4439-96e9-adaec2684196") : secret "dns-default-metrics-tls" not found Jul 23 17:58:14.718747 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:14.718703 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:14.719155 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:14.718878 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:58:14.719155 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:14.718944 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls podName:ce29f6ec-8310-4439-96e9-adaec2684196 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:22.718927127 +0000 UTC m=+52.160425701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls") pod "dns-default-5nhzx" (UID: "ce29f6ec-8310-4439-96e9-adaec2684196") : secret "dns-default-metrics-tls" not found Jul 23 17:58:22.776688 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:22.776644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:22.777176 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:22.776798 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:58:22.777176 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:22.776877 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls podName:ce29f6ec-8310-4439-96e9-adaec2684196 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:38.77686188 +0000 UTC m=+68.218360454 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls") pod "dns-default-5nhzx" (UID: "ce29f6ec-8310-4439-96e9-adaec2684196") : secret "dns-default-metrics-tls" not found Jul 23 17:58:32.368881 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:32.368850 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rfsgl" Jul 23 17:58:35.760594 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:35.760539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs\") pod \"network-metrics-daemon-lg9sf\" (UID: \"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26\") " pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:58:35.764234 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:35.764216 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 17:58:35.771336 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:35.771318 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 23 17:58:35.771393 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:35.771383 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs podName:b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:39.771368389 +0000 UTC m=+129.212866961 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs") pod "network-metrics-daemon-lg9sf" (UID: "b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26") : secret "metrics-daemon-secret" not found Jul 23 17:58:35.860919 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:35.860887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:58:35.865318 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:35.865301 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 17:58:35.874783 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:35.874753 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d669e26-a0e5-4073-8f2c-64b52d9f959d-original-pull-secret\") pod \"global-pull-secret-syncer-865dp\" (UID: \"1d669e26-a0e5-4073-8f2c-64b52d9f959d\") " pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:58:35.924124 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:35.924084 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-865dp" Jul 23 17:58:35.962422 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:35.962388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjf7\" (UniqueName: \"kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7\") pod \"network-check-target-tcz9v\" (UID: \"d3acdd92-76f4-4cf9-b91e-4ba8016f441a\") " pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:58:35.966029 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:35.966009 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 17:58:35.976667 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:35.976640 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 17:58:35.987422 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:35.987359 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhjf7\" (UniqueName: \"kubernetes.io/projected/d3acdd92-76f4-4cf9-b91e-4ba8016f441a-kube-api-access-nhjf7\") pod \"network-check-target-tcz9v\" (UID: \"d3acdd92-76f4-4cf9-b91e-4ba8016f441a\") " pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:58:36.048626 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:36.048549 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-865dp"] Jul 23 17:58:36.051167 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:58:36.051141 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d669e26_a0e5_4073_8f2c_64b52d9f959d.slice/crio-d1f1f6ceaf9a7110640a4ffb4e261a45abf2d0ec99f3f87eb3f4fd4d01acfec2 WatchSource:0}: Error finding container d1f1f6ceaf9a7110640a4ffb4e261a45abf2d0ec99f3f87eb3f4fd4d01acfec2: Status 404 returned error can't find the container with id d1f1f6ceaf9a7110640a4ffb4e261a45abf2d0ec99f3f87eb3f4fd4d01acfec2 Jul 23 17:58:36.232635 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:36.232603 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qd4dg\"" Jul 23 17:58:36.240228 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:36.240212 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:58:36.360889 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:36.360858 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tcz9v"] Jul 23 17:58:36.363569 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:58:36.363545 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3acdd92_76f4_4cf9_b91e_4ba8016f441a.slice/crio-7fa8c91b46ffb7e51f4fa12d696ccdfce56d2580b0ebc9509d4f793bf4e9e433 WatchSource:0}: Error finding container 7fa8c91b46ffb7e51f4fa12d696ccdfce56d2580b0ebc9509d4f793bf4e9e433: Status 404 returned error can't find the container with id 7fa8c91b46ffb7e51f4fa12d696ccdfce56d2580b0ebc9509d4f793bf4e9e433 Jul 23 17:58:36.441752 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:36.441713 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tcz9v" event={"ID":"d3acdd92-76f4-4cf9-b91e-4ba8016f441a","Type":"ContainerStarted","Data":"7fa8c91b46ffb7e51f4fa12d696ccdfce56d2580b0ebc9509d4f793bf4e9e433"} Jul 23 17:58:36.442697 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:36.442673 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-865dp" event={"ID":"1d669e26-a0e5-4073-8f2c-64b52d9f959d","Type":"ContainerStarted","Data":"d1f1f6ceaf9a7110640a4ffb4e261a45abf2d0ec99f3f87eb3f4fd4d01acfec2"} Jul 23 17:58:38.788872 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:38.788824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:58:38.789270 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:38.789006 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:58:38.789270 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:38.789074 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls podName:ce29f6ec-8310-4439-96e9-adaec2684196 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:10.789060098 +0000 UTC m=+100.230558676 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls") pod "dns-default-5nhzx" (UID: "ce29f6ec-8310-4439-96e9-adaec2684196") : secret "dns-default-metrics-tls" not found Jul 23 17:58:41.456720 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.456627 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tcz9v" event={"ID":"d3acdd92-76f4-4cf9-b91e-4ba8016f441a","Type":"ContainerStarted","Data":"81ec94de78a9fe8105b9a42e91f1145708480efdd280acbb3a3f149b53a7d2fe"} Jul 23 17:58:41.457188 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.456741 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:58:41.458065 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.458039 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-865dp" event={"ID":"1d669e26-a0e5-4073-8f2c-64b52d9f959d","Type":"ContainerStarted","Data":"8e17ef636cbf4b740608991abf7f5d308e4929da186f2deca34261d7007871cd"} Jul 23 17:58:41.480097 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.480054 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tcz9v" podStartSLOduration=65.657044884 podStartE2EDuration="1m10.480040404s" totalImagesPullingTime="4.82299552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:58:41.479515155 +0000 UTC m=+70.921013768" watchObservedRunningTime="2026-07-23 17:58:41.480040404 +0000 UTC m=+70.921539028" Jul 23 17:58:41.498289 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.498245 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-865dp" podStartSLOduration=64.359519828 podStartE2EDuration="1m9.498232518s" totalImagesPullingTime="5.13871269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:58:41.49724935 +0000 UTC m=+70.938747944" watchObservedRunningTime="2026-07-23 17:58:41.498232518 +0000 UTC m=+70.939731157" Jul 23 17:58:41.796872 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.796835 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n77rj"] Jul 23 17:58:41.798532 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.798516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n77rj" Jul 23 17:58:41.801260 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.801239 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tdskh\"" Jul 23 17:58:41.801420 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.801404 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 23 17:58:41.802254 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.802233 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 23 17:58:41.814568 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.814544 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n77rj"] Jul 23 17:58:41.893931 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.893899 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6"] Jul 23 17:58:41.895910 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.895887 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" Jul 23 17:58:41.897332 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.897307 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-944979c7d-2pgzv"] Jul 23 17:58:41.898542 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.898516 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 23 17:58:41.898542 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.898521 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 23 17:58:41.898688 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.898556 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 23 17:58:41.898899 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.898882 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:41.898998 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.898983 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9crtf\"" Jul 23 17:58:41.899415 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.899399 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 23 17:58:41.902026 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.902007 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 17:58:41.902159 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.902137 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 23 17:58:41.902232 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.902165 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 23 17:58:41.902232 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.902183 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 23 17:58:41.902596 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.902489 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 17:58:41.902596 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.902494 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 23 17:58:41.903251 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.903236 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-l2z64\"" Jul 23 17:58:41.909888 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.909869 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6"] Jul 23 17:58:41.911270 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.911250 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzmxq\" (UniqueName: \"kubernetes.io/projected/d49329fa-49a2-4376-82d6-aa2b7830f925-kube-api-access-tzmxq\") pod \"volume-data-source-validator-78cf4c6f6-n77rj\" (UID: \"d49329fa-49a2-4376-82d6-aa2b7830f925\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n77rj" Jul 23 17:58:41.911407 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.911280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmk55\" (UniqueName: \"kubernetes.io/projected/97ce2dab-10a1-4fa6-818d-ec1aa4586798-kube-api-access-zmk55\") pod \"kube-storage-version-migrator-operator-7658d9b749-8bqz6\" (UID: \"97ce2dab-10a1-4fa6-818d-ec1aa4586798\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" Jul 23 17:58:41.911407 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.911393 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97ce2dab-10a1-4fa6-818d-ec1aa4586798-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-8bqz6\" (UID: \"97ce2dab-10a1-4fa6-818d-ec1aa4586798\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" Jul 23 17:58:41.911490 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.911417 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97ce2dab-10a1-4fa6-818d-ec1aa4586798-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-8bqz6\" (UID: \"97ce2dab-10a1-4fa6-818d-ec1aa4586798\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" Jul 23 17:58:41.914257 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:41.914238 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-944979c7d-2pgzv"] Jul 23 17:58:42.012134 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.012093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-default-certificate\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.012306 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.012168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97ce2dab-10a1-4fa6-818d-ec1aa4586798-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-8bqz6\" (UID: \"97ce2dab-10a1-4fa6-818d-ec1aa4586798\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" Jul 23 17:58:42.012306 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.012191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97ce2dab-10a1-4fa6-818d-ec1aa4586798-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-8bqz6\" (UID: \"97ce2dab-10a1-4fa6-818d-ec1aa4586798\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" Jul 23 17:58:42.012306 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.012211 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.012306 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.012235 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzmxq\" (UniqueName: \"kubernetes.io/projected/d49329fa-49a2-4376-82d6-aa2b7830f925-kube-api-access-tzmxq\") pod \"volume-data-source-validator-78cf4c6f6-n77rj\" (UID: \"d49329fa-49a2-4376-82d6-aa2b7830f925\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n77rj" Jul 23 17:58:42.012306 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.012253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmk55\" (UniqueName: \"kubernetes.io/projected/97ce2dab-10a1-4fa6-818d-ec1aa4586798-kube-api-access-zmk55\") pod \"kube-storage-version-migrator-operator-7658d9b749-8bqz6\" (UID: \"97ce2dab-10a1-4fa6-818d-ec1aa4586798\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" Jul 23 17:58:42.012306 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.012273 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.012306 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.012296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw6zm\" (UniqueName: \"kubernetes.io/projected/1fbd69ae-ab96-406e-97c8-d36b6c314b83-kube-api-access-sw6zm\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.012558 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.012339 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-stats-auth\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.012746 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.012716 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97ce2dab-10a1-4fa6-818d-ec1aa4586798-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-8bqz6\" (UID: \"97ce2dab-10a1-4fa6-818d-ec1aa4586798\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" Jul 23 17:58:42.014488 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.014463 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97ce2dab-10a1-4fa6-818d-ec1aa4586798-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-8bqz6\" (UID: \"97ce2dab-10a1-4fa6-818d-ec1aa4586798\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" Jul 23 17:58:42.022413 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.022386 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmk55\" (UniqueName: \"kubernetes.io/projected/97ce2dab-10a1-4fa6-818d-ec1aa4586798-kube-api-access-zmk55\") pod \"kube-storage-version-migrator-operator-7658d9b749-8bqz6\" (UID: \"97ce2dab-10a1-4fa6-818d-ec1aa4586798\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" Jul 23 17:58:42.022819 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.022781 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzmxq\" (UniqueName: \"kubernetes.io/projected/d49329fa-49a2-4376-82d6-aa2b7830f925-kube-api-access-tzmxq\") pod \"volume-data-source-validator-78cf4c6f6-n77rj\" (UID: \"d49329fa-49a2-4376-82d6-aa2b7830f925\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n77rj" Jul 23 17:58:42.107308 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.107222 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n77rj" Jul 23 17:58:42.113636 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.113614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.113733 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.113650 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.113733 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.113677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw6zm\" (UniqueName: \"kubernetes.io/projected/1fbd69ae-ab96-406e-97c8-d36b6c314b83-kube-api-access-sw6zm\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.113849 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.113757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-stats-auth\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.113849 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:42.113831 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle podName:1fbd69ae-ab96-406e-97c8-d36b6c314b83 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:42.613794229 +0000 UTC m=+72.055292833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle") pod "router-default-944979c7d-2pgzv" (UID: "1fbd69ae-ab96-406e-97c8-d36b6c314b83") : configmap references non-existent config key: service-ca.crt Jul 23 17:58:42.114142 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.113859 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-default-certificate\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.114142 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:42.113864 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 17:58:42.114142 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:42.113925 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs podName:1fbd69ae-ab96-406e-97c8-d36b6c314b83 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:42.613916334 +0000 UTC m=+72.055414906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs") pod "router-default-944979c7d-2pgzv" (UID: "1fbd69ae-ab96-406e-97c8-d36b6c314b83") : secret "router-metrics-certs-default" not found Jul 23 17:58:42.116221 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.116198 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-stats-auth\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.116422 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.116405 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-default-certificate\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.123217 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.123193 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw6zm\" (UniqueName: \"kubernetes.io/projected/1fbd69ae-ab96-406e-97c8-d36b6c314b83-kube-api-access-sw6zm\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.206338 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.206297 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" Jul 23 17:58:42.228052 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.228018 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n77rj"] Jul 23 17:58:42.230911 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:58:42.230880 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49329fa_49a2_4376_82d6_aa2b7830f925.slice/crio-3e8dcde1ad927a136ca8838a60b736545a34903959156e226c167cb22dc4d19d WatchSource:0}: Error finding container 3e8dcde1ad927a136ca8838a60b736545a34903959156e226c167cb22dc4d19d: Status 404 returned error can't find the container with id 3e8dcde1ad927a136ca8838a60b736545a34903959156e226c167cb22dc4d19d Jul 23 17:58:42.330506 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.330479 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6"] Jul 23 17:58:42.333586 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:58:42.333559 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ce2dab_10a1_4fa6_818d_ec1aa4586798.slice/crio-2d1663662c760fbe4f8fb197d0575482b30ed52670b56526bb95d4b3231f8f99 WatchSource:0}: Error finding container 2d1663662c760fbe4f8fb197d0575482b30ed52670b56526bb95d4b3231f8f99: Status 404 returned error can't find the container with id 2d1663662c760fbe4f8fb197d0575482b30ed52670b56526bb95d4b3231f8f99 Jul 23 17:58:42.461896 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.461800 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" event={"ID":"97ce2dab-10a1-4fa6-818d-ec1aa4586798","Type":"ContainerStarted","Data":"2d1663662c760fbe4f8fb197d0575482b30ed52670b56526bb95d4b3231f8f99"} Jul 23 17:58:42.462835 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.462792 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n77rj" event={"ID":"d49329fa-49a2-4376-82d6-aa2b7830f925","Type":"ContainerStarted","Data":"3e8dcde1ad927a136ca8838a60b736545a34903959156e226c167cb22dc4d19d"} Jul 23 17:58:42.618760 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.618721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.618979 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:42.618836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:42.618979 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:42.618917 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle podName:1fbd69ae-ab96-406e-97c8-d36b6c314b83 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:43.618895754 +0000 UTC m=+73.060394343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle") pod "router-default-944979c7d-2pgzv" (UID: "1fbd69ae-ab96-406e-97c8-d36b6c314b83") : configmap references non-existent config key: service-ca.crt Jul 23 17:58:42.618979 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:42.618945 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 17:58:42.619114 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:42.618997 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs podName:1fbd69ae-ab96-406e-97c8-d36b6c314b83 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:43.618981192 +0000 UTC m=+73.060479768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs") pod "router-default-944979c7d-2pgzv" (UID: "1fbd69ae-ab96-406e-97c8-d36b6c314b83") : secret "router-metrics-certs-default" not found Jul 23 17:58:43.627250 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:43.627210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:43.627250 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:43.627257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:43.627734 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:43.627389 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 17:58:43.627734 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:43.627395 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle podName:1fbd69ae-ab96-406e-97c8-d36b6c314b83 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:45.627379591 +0000 UTC m=+75.068878166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle") pod "router-default-944979c7d-2pgzv" (UID: "1fbd69ae-ab96-406e-97c8-d36b6c314b83") : configmap references non-existent config key: service-ca.crt Jul 23 17:58:43.627734 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:43.627471 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs podName:1fbd69ae-ab96-406e-97c8-d36b6c314b83 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:45.627449044 +0000 UTC m=+75.068947630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs") pod "router-default-944979c7d-2pgzv" (UID: "1fbd69ae-ab96-406e-97c8-d36b6c314b83") : secret "router-metrics-certs-default" not found Jul 23 17:58:44.472653 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:44.472606 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n77rj" event={"ID":"d49329fa-49a2-4376-82d6-aa2b7830f925","Type":"ContainerStarted","Data":"e8863a5f9da97ad0dcaa43e96800d0819cd0d2dde3387d355c51eeccebfb2947"} Jul 23 17:58:45.476436 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:45.476402 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" event={"ID":"97ce2dab-10a1-4fa6-818d-ec1aa4586798","Type":"ContainerStarted","Data":"2f109f6f374511b94f5118e29315013dee796c5bfde50cdcb714bffa278ae5e4"} Jul 23 17:58:45.496566 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:45.496517 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-n77rj" podStartSLOduration=3.317206258 podStartE2EDuration="4.496502511s" totalImagesPullingTime="1.179296253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:58:44.49415998 +0000 UTC m=+73.935658575" watchObservedRunningTime="2026-07-23 17:58:45.496502511 +0000 UTC m=+74.938001104" Jul 23 17:58:45.646600 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:45.646565 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:45.646600 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:45.646606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:45.646781 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:45.646713 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 17:58:45.646781 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:45.646736 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle podName:1fbd69ae-ab96-406e-97c8-d36b6c314b83 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:49.646720394 +0000 UTC m=+79.088218968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle") pod "router-default-944979c7d-2pgzv" (UID: "1fbd69ae-ab96-406e-97c8-d36b6c314b83") : configmap references non-existent config key: service-ca.crt Jul 23 17:58:45.646899 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:45.646821 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs podName:1fbd69ae-ab96-406e-97c8-d36b6c314b83 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:49.646792371 +0000 UTC m=+79.088290943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs") pod "router-default-944979c7d-2pgzv" (UID: "1fbd69ae-ab96-406e-97c8-d36b6c314b83") : secret "router-metrics-certs-default" not found Jul 23 17:58:47.465713 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.465657 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" podStartSLOduration=4.09595694 podStartE2EDuration="6.465640479s" totalImagesPullingTime="2.369683539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:58:45.496487704 +0000 UTC m=+74.937986299" watchObservedRunningTime="2026-07-23 17:58:47.465640479 +0000 UTC m=+76.907139075" Jul 23 17:58:47.466510 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.466490 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-krgbq"] Jul 23 17:58:47.469104 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.469088 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:47.474363 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.474342 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dj9xd\"" Jul 23 17:58:47.474529 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.474513 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 17:58:47.477360 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.477343 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 17:58:47.477894 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.477880 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 17:58:47.481250 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.481221 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 17:58:47.501098 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.501074 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-krgbq"] Jul 23 17:58:47.561057 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.561021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:47.561057 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.561061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e46a507c-b10b-4bac-8e85-01475e644526-data-volume\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:47.561286 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.561093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e46a507c-b10b-4bac-8e85-01475e644526-crio-socket\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:47.561286 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.561186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdh4g\" (UniqueName: \"kubernetes.io/projected/e46a507c-b10b-4bac-8e85-01475e644526-kube-api-access-rdh4g\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:47.561286 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.561224 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:47.662503 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.662466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:47.662671 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.662521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:47.662671 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.662544 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e46a507c-b10b-4bac-8e85-01475e644526-data-volume\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:47.662671 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:47.662622 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 17:58:47.662671 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:47.662632 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 17:58:47.662671 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.662649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e46a507c-b10b-4bac-8e85-01475e644526-crio-socket\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:47.662961 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:47.662694 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm podName:e46a507c-b10b-4bac-8e85-01475e644526 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:48.16267528 +0000 UTC m=+77.604173870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm") pod "insights-runtime-extractor-krgbq" (UID: "e46a507c-b10b-4bac-8e85-01475e644526") : configmap "kube-rbac-proxy" not found Jul 23 17:58:47.662961 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.662715 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e46a507c-b10b-4bac-8e85-01475e644526-crio-socket\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:47.662961 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:47.662733 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls podName:e46a507c-b10b-4bac-8e85-01475e644526 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:48.162711521 +0000 UTC m=+77.604210097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls") pod "insights-runtime-extractor-krgbq" (UID: "e46a507c-b10b-4bac-8e85-01475e644526") : secret "insights-runtime-extractor-tls" not found Jul 23 17:58:47.662961 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.662847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdh4g\" (UniqueName: \"kubernetes.io/projected/e46a507c-b10b-4bac-8e85-01475e644526-kube-api-access-rdh4g\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:47.662961 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.662851 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e46a507c-b10b-4bac-8e85-01475e644526-data-volume\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:47.707152 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:47.707117 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdh4g\" (UniqueName: \"kubernetes.io/projected/e46a507c-b10b-4bac-8e85-01475e644526-kube-api-access-rdh4g\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:48.165672 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:48.165628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:48.165880 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:48.165684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:48.165880 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:48.165753 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 17:58:48.165880 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:48.165779 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 17:58:48.165880 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:48.165833 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm podName:e46a507c-b10b-4bac-8e85-01475e644526 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:49.165800231 +0000 UTC m=+78.607298806 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm") pod "insights-runtime-extractor-krgbq" (UID: "e46a507c-b10b-4bac-8e85-01475e644526") : configmap "kube-rbac-proxy" not found Jul 23 17:58:48.165880 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:48.165849 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls podName:e46a507c-b10b-4bac-8e85-01475e644526 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:49.165843741 +0000 UTC m=+78.607342316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls") pod "insights-runtime-extractor-krgbq" (UID: "e46a507c-b10b-4bac-8e85-01475e644526") : secret "insights-runtime-extractor-tls" not found Jul 23 17:58:49.171873 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.171825 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:49.172277 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.171895 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:49.172277 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:49.171970 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 17:58:49.172277 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:49.172044 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm podName:e46a507c-b10b-4bac-8e85-01475e644526 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:51.172023444 +0000 UTC m=+80.613522020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm") pod "insights-runtime-extractor-krgbq" (UID: "e46a507c-b10b-4bac-8e85-01475e644526") : configmap "kube-rbac-proxy" not found Jul 23 17:58:49.172277 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:49.172050 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 17:58:49.172277 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:49.172096 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls podName:e46a507c-b10b-4bac-8e85-01475e644526 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:51.172083355 +0000 UTC m=+80.613581930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls") pod "insights-runtime-extractor-krgbq" (UID: "e46a507c-b10b-4bac-8e85-01475e644526") : secret "insights-runtime-extractor-tls" not found Jul 23 17:58:49.675232 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.675197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:49.675232 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.675239 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:49.675500 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:49.675376 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 17:58:49.675500 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:49.675393 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle podName:1fbd69ae-ab96-406e-97c8-d36b6c314b83 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:57.675380693 +0000 UTC m=+87.116879269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle") pod "router-default-944979c7d-2pgzv" (UID: "1fbd69ae-ab96-406e-97c8-d36b6c314b83") : configmap references non-existent config key: service-ca.crt Jul 23 17:58:49.675500 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:49.675440 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs podName:1fbd69ae-ab96-406e-97c8-d36b6c314b83 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:57.67542297 +0000 UTC m=+87.116921556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs") pod "router-default-944979c7d-2pgzv" (UID: "1fbd69ae-ab96-406e-97c8-d36b6c314b83") : secret "router-metrics-certs-default" not found Jul 23 17:58:49.752442 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.752412 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-9ndck"] Jul 23 17:58:49.754422 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.754405 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-9ndck" Jul 23 17:58:49.761368 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.761341 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-88cns\"" Jul 23 17:58:49.761453 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.761342 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 23 17:58:49.762607 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.762586 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 23 17:58:49.762708 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.762648 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 23 17:58:49.762708 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.762664 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 23 17:58:49.769064 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.769038 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-9ndck"] Jul 23 17:58:49.876889 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.876852 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs4qx\" (UniqueName: \"kubernetes.io/projected/fb9ae910-e939-4bfb-b068-5601b58f0394-kube-api-access-bs4qx\") pod \"service-ca-58598695f7-9ndck\" (UID: \"fb9ae910-e939-4bfb-b068-5601b58f0394\") " pod="openshift-service-ca/service-ca-58598695f7-9ndck" Jul 23 17:58:49.877093 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.876953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fb9ae910-e939-4bfb-b068-5601b58f0394-signing-cabundle\") pod \"service-ca-58598695f7-9ndck\" (UID: \"fb9ae910-e939-4bfb-b068-5601b58f0394\") " pod="openshift-service-ca/service-ca-58598695f7-9ndck" Jul 23 17:58:49.877093 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.876994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fb9ae910-e939-4bfb-b068-5601b58f0394-signing-key\") pod \"service-ca-58598695f7-9ndck\" (UID: \"fb9ae910-e939-4bfb-b068-5601b58f0394\") " pod="openshift-service-ca/service-ca-58598695f7-9ndck" Jul 23 17:58:49.978048 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.978014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fb9ae910-e939-4bfb-b068-5601b58f0394-signing-cabundle\") pod \"service-ca-58598695f7-9ndck\" (UID: \"fb9ae910-e939-4bfb-b068-5601b58f0394\") " pod="openshift-service-ca/service-ca-58598695f7-9ndck" Jul 23 17:58:49.978202 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.978064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fb9ae910-e939-4bfb-b068-5601b58f0394-signing-key\") pod \"service-ca-58598695f7-9ndck\" (UID: \"fb9ae910-e939-4bfb-b068-5601b58f0394\") " pod="openshift-service-ca/service-ca-58598695f7-9ndck" Jul 23 17:58:49.978202 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.978150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs4qx\" (UniqueName: \"kubernetes.io/projected/fb9ae910-e939-4bfb-b068-5601b58f0394-kube-api-access-bs4qx\") pod \"service-ca-58598695f7-9ndck\" (UID: \"fb9ae910-e939-4bfb-b068-5601b58f0394\") " pod="openshift-service-ca/service-ca-58598695f7-9ndck" Jul 23 17:58:49.978667 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.978646 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fb9ae910-e939-4bfb-b068-5601b58f0394-signing-cabundle\") pod \"service-ca-58598695f7-9ndck\" (UID: \"fb9ae910-e939-4bfb-b068-5601b58f0394\") " pod="openshift-service-ca/service-ca-58598695f7-9ndck" Jul 23 17:58:49.980405 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.980380 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fb9ae910-e939-4bfb-b068-5601b58f0394-signing-key\") pod \"service-ca-58598695f7-9ndck\" (UID: \"fb9ae910-e939-4bfb-b068-5601b58f0394\") " pod="openshift-service-ca/service-ca-58598695f7-9ndck" Jul 23 17:58:49.991863 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:49.991839 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs4qx\" (UniqueName: \"kubernetes.io/projected/fb9ae910-e939-4bfb-b068-5601b58f0394-kube-api-access-bs4qx\") pod \"service-ca-58598695f7-9ndck\" (UID: \"fb9ae910-e939-4bfb-b068-5601b58f0394\") " pod="openshift-service-ca/service-ca-58598695f7-9ndck" Jul 23 17:58:50.063705 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:50.063671 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-9ndck" Jul 23 17:58:50.193224 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:50.193194 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-9ndck"] Jul 23 17:58:50.196397 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:58:50.196367 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9ae910_e939_4bfb_b068_5601b58f0394.slice/crio-22c8a4bb686efbda0e24f7bcc85524556d4fc23247c53eefe5b3c023f3504027 WatchSource:0}: Error finding container 22c8a4bb686efbda0e24f7bcc85524556d4fc23247c53eefe5b3c023f3504027: Status 404 returned error can't find the container with id 22c8a4bb686efbda0e24f7bcc85524556d4fc23247c53eefe5b3c023f3504027 Jul 23 17:58:50.258088 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:50.258012 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gwvmh_d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad/dns-node-resolver/0.log" Jul 23 17:58:50.492887 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:50.492854 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-9ndck" event={"ID":"fb9ae910-e939-4bfb-b068-5601b58f0394","Type":"ContainerStarted","Data":"22c8a4bb686efbda0e24f7bcc85524556d4fc23247c53eefe5b3c023f3504027"} Jul 23 17:58:51.188100 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:51.188070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:51.188257 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:51.188113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:51.188257 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:51.188186 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 17:58:51.188257 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:51.188207 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 17:58:51.188257 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:51.188248 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm podName:e46a507c-b10b-4bac-8e85-01475e644526 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:55.188230116 +0000 UTC m=+84.629728700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm") pod "insights-runtime-extractor-krgbq" (UID: "e46a507c-b10b-4bac-8e85-01475e644526") : configmap "kube-rbac-proxy" not found Jul 23 17:58:51.188390 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:51.188263 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls podName:e46a507c-b10b-4bac-8e85-01475e644526 nodeName:}" failed. No retries permitted until 2026-07-23 17:58:55.188256275 +0000 UTC m=+84.629754846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls") pod "insights-runtime-extractor-krgbq" (UID: "e46a507c-b10b-4bac-8e85-01475e644526") : secret "insights-runtime-extractor-tls" not found Jul 23 17:58:51.454244 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:51.454173 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x82j5_6098df05-0cf3-46ae-83ea-f16c1edf1260/node-ca/0.log" Jul 23 17:58:52.046212 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:52.046188 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hvkrn_dee23d54-ccac-4849-8abd-3f25a806ca2e/serve-healthcheck-canary/0.log" Jul 23 17:58:52.507566 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:52.507531 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-9ndck" event={"ID":"fb9ae910-e939-4bfb-b068-5601b58f0394","Type":"ContainerStarted","Data":"a53ef561412af1d60b3aba7447d00b5f1c4938deab9339d8e16e146ccbb748b3"} Jul 23 17:58:52.845693 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:52.845611 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-8bqz6_97ce2dab-10a1-4fa6-818d-ec1aa4586798/kube-storage-version-migrator-operator/0.log" Jul 23 17:58:55.226690 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:55.226539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:55.226690 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:55.226614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:58:55.226690 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:55.226669 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 17:58:55.226690 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:55.226696 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 17:58:55.227222 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:55.226734 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm podName:e46a507c-b10b-4bac-8e85-01475e644526 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:03.226718402 +0000 UTC m=+92.668216975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm") pod "insights-runtime-extractor-krgbq" (UID: "e46a507c-b10b-4bac-8e85-01475e644526") : configmap "kube-rbac-proxy" not found Jul 23 17:58:55.227222 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:55.226748 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls podName:e46a507c-b10b-4bac-8e85-01475e644526 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:03.22674211 +0000 UTC m=+92.668240682 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls") pod "insights-runtime-extractor-krgbq" (UID: "e46a507c-b10b-4bac-8e85-01475e644526") : secret "insights-runtime-extractor-tls" not found Jul 23 17:58:57.746152 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:57.746116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:57.746561 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:58:57.746162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:58:57.746561 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:57.746265 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 17:58:57.746561 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:57.746329 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs podName:1fbd69ae-ab96-406e-97c8-d36b6c314b83 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:13.74631287 +0000 UTC m=+103.187811445 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs") pod "router-default-944979c7d-2pgzv" (UID: "1fbd69ae-ab96-406e-97c8-d36b6c314b83") : secret "router-metrics-certs-default" not found Jul 23 17:58:57.746561 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:58:57.746346 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle podName:1fbd69ae-ab96-406e-97c8-d36b6c314b83 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:13.746338766 +0000 UTC m=+103.187837347 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle") pod "router-default-944979c7d-2pgzv" (UID: "1fbd69ae-ab96-406e-97c8-d36b6c314b83") : configmap references non-existent config key: service-ca.crt Jul 23 17:59:03.292541 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:03.292500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:59:03.292960 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:03.292618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:59:03.292960 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:59:03.292667 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 17:59:03.292960 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:59:03.292737 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls podName:e46a507c-b10b-4bac-8e85-01475e644526 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:19.292719126 +0000 UTC m=+108.734217715 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls") pod "insights-runtime-extractor-krgbq" (UID: "e46a507c-b10b-4bac-8e85-01475e644526") : secret "insights-runtime-extractor-tls" not found Jul 23 17:59:03.293206 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:03.293186 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e46a507c-b10b-4bac-8e85-01475e644526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:59:10.856988 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:10.856954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:59:10.859295 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:10.859272 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce29f6ec-8310-4439-96e9-adaec2684196-metrics-tls\") pod \"dns-default-5nhzx\" (UID: \"ce29f6ec-8310-4439-96e9-adaec2684196\") " pod="openshift-dns/dns-default-5nhzx" Jul 23 17:59:11.120196 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.120121 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-djwnl\"" Jul 23 17:59:11.127712 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.127693 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5nhzx" Jul 23 17:59:11.261919 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.261865 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-9ndck" podStartSLOduration=20.451157118 podStartE2EDuration="22.261849367s" totalImagesPullingTime="1.810692249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:58:52.53693814 +0000 UTC m=+81.978436753" watchObservedRunningTime="2026-07-23 17:59:11.261849367 +0000 UTC m=+100.703347961" Jul 23 17:59:11.262071 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.261989 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5nhzx"] Jul 23 17:59:11.264886 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:11.264864 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce29f6ec_8310_4439_96e9_adaec2684196.slice/crio-db67e85ef462fadbdb5813a5dde0bbd21a78f81735a76e52f25ec2095f004be8 WatchSource:0}: Error finding container db67e85ef462fadbdb5813a5dde0bbd21a78f81735a76e52f25ec2095f004be8: Status 404 returned error can't find the container with id db67e85ef462fadbdb5813a5dde0bbd21a78f81735a76e52f25ec2095f004be8 Jul 23 17:59:11.562533 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.562505 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5nhzx" event={"ID":"ce29f6ec-8310-4439-96e9-adaec2684196","Type":"ContainerStarted","Data":"db67e85ef462fadbdb5813a5dde0bbd21a78f81735a76e52f25ec2095f004be8"} Jul 23 17:59:11.757714 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.757677 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-d997q"] Jul 23 17:59:11.759910 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.759889 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-d997q" Jul 23 17:59:11.765666 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.765643 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-c5dct\"" Jul 23 17:59:11.765769 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.765643 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 23 17:59:11.765769 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.765644 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 23 17:59:11.864352 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.864265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d178632a-a6f8-4d69-bf47-a4302711b7e9-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-d997q\" (UID: \"d178632a-a6f8-4d69-bf47-a4302711b7e9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-d997q" Jul 23 17:59:11.864352 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.864326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d178632a-a6f8-4d69-bf47-a4302711b7e9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-d997q\" (UID: \"d178632a-a6f8-4d69-bf47-a4302711b7e9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-d997q" Jul 23 17:59:11.887934 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.887906 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-d997q"] Jul 23 17:59:11.964798 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.964758 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d178632a-a6f8-4d69-bf47-a4302711b7e9-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-d997q\" (UID: \"d178632a-a6f8-4d69-bf47-a4302711b7e9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-d997q" Jul 23 17:59:11.965011 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.964874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d178632a-a6f8-4d69-bf47-a4302711b7e9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-d997q\" (UID: \"d178632a-a6f8-4d69-bf47-a4302711b7e9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-d997q" Jul 23 17:59:11.965524 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.965491 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d178632a-a6f8-4d69-bf47-a4302711b7e9-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-d997q\" (UID: \"d178632a-a6f8-4d69-bf47-a4302711b7e9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-d997q" Jul 23 17:59:11.967611 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:11.967581 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d178632a-a6f8-4d69-bf47-a4302711b7e9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-d997q\" (UID: \"d178632a-a6f8-4d69-bf47-a4302711b7e9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-d997q" Jul 23 17:59:12.068574 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.068545 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-d997q" Jul 23 17:59:12.098135 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.098094 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhjwd"] Jul 23 17:59:12.100236 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.100215 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhjwd" Jul 23 17:59:12.136139 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.136106 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2wpnd\"" Jul 23 17:59:12.136680 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.136652 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx"] Jul 23 17:59:12.139459 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.138988 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" Jul 23 17:59:12.142387 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.142331 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhjwd"] Jul 23 17:59:12.149825 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.149657 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 23 17:59:12.149825 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.149693 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 23 17:59:12.150035 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.150001 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 23 17:59:12.151090 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.151009 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx"] Jul 23 17:59:12.154696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.154677 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 23 17:59:12.267428 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.267396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/91b7eda8-056a-4c20-9923-eb26f0479ea7-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f86564f98-49kzx\" (UID: \"91b7eda8-056a-4c20-9923-eb26f0479ea7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" Jul 23 17:59:12.267428 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.267434 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99xl5\" (UniqueName: \"kubernetes.io/projected/91b7eda8-056a-4c20-9923-eb26f0479ea7-kube-api-access-99xl5\") pod \"klusterlet-addon-workmgr-6f86564f98-49kzx\" (UID: \"91b7eda8-056a-4c20-9923-eb26f0479ea7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" Jul 23 17:59:12.267723 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.267481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbl5z\" (UniqueName: \"kubernetes.io/projected/24ab2fca-697a-4148-bf47-9bd1c48eb5a9-kube-api-access-gbl5z\") pod \"network-check-source-7bc5b6bf85-lhjwd\" (UID: \"24ab2fca-697a-4148-bf47-9bd1c48eb5a9\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhjwd" Jul 23 17:59:12.267723 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.267591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91b7eda8-056a-4c20-9923-eb26f0479ea7-tmp\") pod \"klusterlet-addon-workmgr-6f86564f98-49kzx\" (UID: \"91b7eda8-056a-4c20-9923-eb26f0479ea7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" Jul 23 17:59:12.285285 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.285251 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-d997q"] Jul 23 17:59:12.295224 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:12.295197 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd178632a_a6f8_4d69_bf47_a4302711b7e9.slice/crio-57670b1adcc5bea51013c9481dfbc12c0934bd48f7a9694592512d55f4a43915 WatchSource:0}: Error finding container 57670b1adcc5bea51013c9481dfbc12c0934bd48f7a9694592512d55f4a43915: Status 404 returned error can't find the container with id 57670b1adcc5bea51013c9481dfbc12c0934bd48f7a9694592512d55f4a43915 Jul 23 17:59:12.368759 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.368719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91b7eda8-056a-4c20-9923-eb26f0479ea7-tmp\") pod \"klusterlet-addon-workmgr-6f86564f98-49kzx\" (UID: \"91b7eda8-056a-4c20-9923-eb26f0479ea7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" Jul 23 17:59:12.368966 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.368823 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/91b7eda8-056a-4c20-9923-eb26f0479ea7-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f86564f98-49kzx\" (UID: \"91b7eda8-056a-4c20-9923-eb26f0479ea7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" Jul 23 17:59:12.368966 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.368930 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99xl5\" (UniqueName: \"kubernetes.io/projected/91b7eda8-056a-4c20-9923-eb26f0479ea7-kube-api-access-99xl5\") pod \"klusterlet-addon-workmgr-6f86564f98-49kzx\" (UID: \"91b7eda8-056a-4c20-9923-eb26f0479ea7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" Jul 23 17:59:12.369081 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.368975 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbl5z\" (UniqueName: \"kubernetes.io/projected/24ab2fca-697a-4148-bf47-9bd1c48eb5a9-kube-api-access-gbl5z\") pod \"network-check-source-7bc5b6bf85-lhjwd\" (UID: \"24ab2fca-697a-4148-bf47-9bd1c48eb5a9\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhjwd" Jul 23 17:59:12.369282 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.369199 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91b7eda8-056a-4c20-9923-eb26f0479ea7-tmp\") pod \"klusterlet-addon-workmgr-6f86564f98-49kzx\" (UID: \"91b7eda8-056a-4c20-9923-eb26f0479ea7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" Jul 23 17:59:12.371791 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.371769 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/91b7eda8-056a-4c20-9923-eb26f0479ea7-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f86564f98-49kzx\" (UID: \"91b7eda8-056a-4c20-9923-eb26f0479ea7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" Jul 23 17:59:12.402455 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.402420 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-db4467ff9-p7t8v"] Jul 23 17:59:12.408421 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.408397 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.426834 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.426793 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 17:59:12.426972 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.426945 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lk5rs\"" Jul 23 17:59:12.426972 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.426967 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 17:59:12.429130 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.429102 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 17:59:12.429251 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.429228 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbl5z\" (UniqueName: \"kubernetes.io/projected/24ab2fca-697a-4148-bf47-9bd1c48eb5a9-kube-api-access-gbl5z\") pod \"network-check-source-7bc5b6bf85-lhjwd\" (UID: \"24ab2fca-697a-4148-bf47-9bd1c48eb5a9\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhjwd" Jul 23 17:59:12.444072 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.444040 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 17:59:12.452170 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.452131 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99xl5\" (UniqueName: \"kubernetes.io/projected/91b7eda8-056a-4c20-9923-eb26f0479ea7-kube-api-access-99xl5\") pod \"klusterlet-addon-workmgr-6f86564f98-49kzx\" (UID: \"91b7eda8-056a-4c20-9923-eb26f0479ea7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" Jul 23 17:59:12.452831 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.452795 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" Jul 23 17:59:12.465775 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.465752 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tcz9v" Jul 23 17:59:12.470698 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.470673 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-db4467ff9-p7t8v"] Jul 23 17:59:12.568420 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.568380 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-d997q" event={"ID":"d178632a-a6f8-4d69-bf47-a4302711b7e9","Type":"ContainerStarted","Data":"57670b1adcc5bea51013c9481dfbc12c0934bd48f7a9694592512d55f4a43915"} Jul 23 17:59:12.569691 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.569669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5431f723-ba76-49a7-ba80-df67a9d7e69b-image-registry-private-configuration\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.569788 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.569734 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlz8b\" (UniqueName: \"kubernetes.io/projected/5431f723-ba76-49a7-ba80-df67a9d7e69b-kube-api-access-qlz8b\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.569871 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.569785 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5431f723-ba76-49a7-ba80-df67a9d7e69b-bound-sa-token\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.569947 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.569929 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5431f723-ba76-49a7-ba80-df67a9d7e69b-registry-certificates\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.570164 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.569976 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5431f723-ba76-49a7-ba80-df67a9d7e69b-trusted-ca\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.570164 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.570022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5431f723-ba76-49a7-ba80-df67a9d7e69b-ca-trust-extracted\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.570164 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.570051 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5431f723-ba76-49a7-ba80-df67a9d7e69b-registry-tls\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.570164 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.570091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5431f723-ba76-49a7-ba80-df67a9d7e69b-installation-pull-secrets\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.671170 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.671076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5431f723-ba76-49a7-ba80-df67a9d7e69b-registry-certificates\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.671170 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.671133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5431f723-ba76-49a7-ba80-df67a9d7e69b-trusted-ca\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.671170 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.671170 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5431f723-ba76-49a7-ba80-df67a9d7e69b-ca-trust-extracted\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.671655 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.671369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5431f723-ba76-49a7-ba80-df67a9d7e69b-registry-tls\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.671655 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.671438 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5431f723-ba76-49a7-ba80-df67a9d7e69b-installation-pull-secrets\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.671655 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.671474 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5431f723-ba76-49a7-ba80-df67a9d7e69b-image-registry-private-configuration\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.671655 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.671522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlz8b\" (UniqueName: \"kubernetes.io/projected/5431f723-ba76-49a7-ba80-df67a9d7e69b-kube-api-access-qlz8b\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.671655 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.671574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5431f723-ba76-49a7-ba80-df67a9d7e69b-bound-sa-token\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.671655 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.671634 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5431f723-ba76-49a7-ba80-df67a9d7e69b-ca-trust-extracted\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.672199 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.672053 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5431f723-ba76-49a7-ba80-df67a9d7e69b-registry-certificates\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.672262 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.672241 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5431f723-ba76-49a7-ba80-df67a9d7e69b-trusted-ca\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.673910 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.673878 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx"] Jul 23 17:59:12.674787 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.674763 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5431f723-ba76-49a7-ba80-df67a9d7e69b-registry-tls\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.674945 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.674816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5431f723-ba76-49a7-ba80-df67a9d7e69b-image-registry-private-configuration\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.675529 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.675509 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5431f723-ba76-49a7-ba80-df67a9d7e69b-installation-pull-secrets\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.677539 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:12.677513 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b7eda8_056a_4c20_9923_eb26f0479ea7.slice/crio-75e53460a605830e026160c6a8fed20c85315fb08a73c53d48238b5aa260b8a7 WatchSource:0}: Error finding container 75e53460a605830e026160c6a8fed20c85315fb08a73c53d48238b5aa260b8a7: Status 404 returned error can't find the container with id 75e53460a605830e026160c6a8fed20c85315fb08a73c53d48238b5aa260b8a7 Jul 23 17:59:12.711205 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.711176 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhjwd" Jul 23 17:59:12.712965 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.712945 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlz8b\" (UniqueName: \"kubernetes.io/projected/5431f723-ba76-49a7-ba80-df67a9d7e69b-kube-api-access-qlz8b\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.717893 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.717867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5431f723-ba76-49a7-ba80-df67a9d7e69b-bound-sa-token\") pod \"image-registry-db4467ff9-p7t8v\" (UID: \"5431f723-ba76-49a7-ba80-df67a9d7e69b\") " pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.719836 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.719793 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:12.903777 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.903277 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhjwd"] Jul 23 17:59:12.927591 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:12.927500 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-db4467ff9-p7t8v"] Jul 23 17:59:13.319195 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:13.319157 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ab2fca_697a_4148_bf47_9bd1c48eb5a9.slice/crio-d8e24fa43f2910cb32589fa4ebd24330cf3406f691bef52b0fff485c58219953 WatchSource:0}: Error finding container d8e24fa43f2910cb32589fa4ebd24330cf3406f691bef52b0fff485c58219953: Status 404 returned error can't find the container with id d8e24fa43f2910cb32589fa4ebd24330cf3406f691bef52b0fff485c58219953 Jul 23 17:59:13.320121 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:13.319796 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5431f723_ba76_49a7_ba80_df67a9d7e69b.slice/crio-c2b00a28543f59617d48427ac8d5e79d86bd29e881ed8c59c507347f2a72664d WatchSource:0}: Error finding container c2b00a28543f59617d48427ac8d5e79d86bd29e881ed8c59c507347f2a72664d: Status 404 returned error can't find the container with id c2b00a28543f59617d48427ac8d5e79d86bd29e881ed8c59c507347f2a72664d Jul 23 17:59:13.573437 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:13.573345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" event={"ID":"5431f723-ba76-49a7-ba80-df67a9d7e69b","Type":"ContainerStarted","Data":"c2b00a28543f59617d48427ac8d5e79d86bd29e881ed8c59c507347f2a72664d"} Jul 23 17:59:13.574637 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:13.574613 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhjwd" event={"ID":"24ab2fca-697a-4148-bf47-9bd1c48eb5a9","Type":"ContainerStarted","Data":"d8e24fa43f2910cb32589fa4ebd24330cf3406f691bef52b0fff485c58219953"} Jul 23 17:59:13.575745 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:13.575722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" event={"ID":"91b7eda8-056a-4c20-9923-eb26f0479ea7","Type":"ContainerStarted","Data":"75e53460a605830e026160c6a8fed20c85315fb08a73c53d48238b5aa260b8a7"} Jul 23 17:59:13.781911 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:13.781874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:59:13.782116 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:13.782045 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:59:13.782539 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:13.782513 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbd69ae-ab96-406e-97c8-d36b6c314b83-service-ca-bundle\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:59:13.784320 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:13.784293 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fbd69ae-ab96-406e-97c8-d36b6c314b83-metrics-certs\") pod \"router-default-944979c7d-2pgzv\" (UID: \"1fbd69ae-ab96-406e-97c8-d36b6c314b83\") " pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:59:14.011733 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.011314 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:59:14.230719 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.229004 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-944979c7d-2pgzv"] Jul 23 17:59:14.233876 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:14.233838 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbd69ae_ab96_406e_97c8_d36b6c314b83.slice/crio-9e646f219eb8d10f00f33802b7bf89f62661d0930eadd9ba07fe198474642ade WatchSource:0}: Error finding container 9e646f219eb8d10f00f33802b7bf89f62661d0930eadd9ba07fe198474642ade: Status 404 returned error can't find the container with id 9e646f219eb8d10f00f33802b7bf89f62661d0930eadd9ba07fe198474642ade Jul 23 17:59:14.584351 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.583902 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-d997q" event={"ID":"d178632a-a6f8-4d69-bf47-a4302711b7e9","Type":"ContainerStarted","Data":"527bf4b3bfdb2144e54346e49883375162ae34cb26b0012f8f06c25a5becd169"} Jul 23 17:59:14.588632 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.588281 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" event={"ID":"5431f723-ba76-49a7-ba80-df67a9d7e69b","Type":"ContainerStarted","Data":"ee4ca7384af69333678278f90c174347d20d5dadef7c9f8e7b4fd6a8cd6f482d"} Jul 23 17:59:14.588632 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.588449 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:14.590645 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.590617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhjwd" event={"ID":"24ab2fca-697a-4148-bf47-9bd1c48eb5a9","Type":"ContainerStarted","Data":"8bd082fe4893eb842140df0faa5ec8f61ece9d6cc1ddd93a8f638648e4209929"} Jul 23 17:59:14.593435 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.593407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5nhzx" event={"ID":"ce29f6ec-8310-4439-96e9-adaec2684196","Type":"ContainerStarted","Data":"bf146c288b9ce4acb341734243ce514e8bf490bbe1b5476619a39e3fdcda3101"} Jul 23 17:59:14.593558 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.593460 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5nhzx" event={"ID":"ce29f6ec-8310-4439-96e9-adaec2684196","Type":"ContainerStarted","Data":"5b337f9919fc7630702db4312dd5097b1ef1a1a6def8ed6b381a8b36e0d7c8bb"} Jul 23 17:59:14.593883 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.593833 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5nhzx" Jul 23 17:59:14.596337 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.596300 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-944979c7d-2pgzv" event={"ID":"1fbd69ae-ab96-406e-97c8-d36b6c314b83","Type":"ContainerStarted","Data":"b7fc946e10bcca2c8bae82534eb9e120f78cdb9b81c0b40f16e4e17e43f4b4d2"} Jul 23 17:59:14.596432 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.596339 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-944979c7d-2pgzv" event={"ID":"1fbd69ae-ab96-406e-97c8-d36b6c314b83","Type":"ContainerStarted","Data":"9e646f219eb8d10f00f33802b7bf89f62661d0930eadd9ba07fe198474642ade"} Jul 23 17:59:14.614823 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.614752 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-d997q" podStartSLOduration=1.965227118 podStartE2EDuration="3.614731621s" totalImagesPullingTime="1.649504503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:14.612919449 +0000 UTC m=+104.054418044" watchObservedRunningTime="2026-07-23 17:59:14.614731621 +0000 UTC m=+104.056230216" Jul 23 17:59:14.639225 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.639173 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5nhzx" podStartSLOduration=65.960448127 podStartE2EDuration="1m8.639155407s" totalImagesPullingTime="2.67870728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:14.637700417 +0000 UTC m=+104.079199010" watchObservedRunningTime="2026-07-23 17:59:14.639155407 +0000 UTC m=+104.080654005" Jul 23 17:59:14.662769 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.662425 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" podStartSLOduration=2.662403143 podStartE2EDuration="2.662403143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:14.661235155 +0000 UTC m=+104.102733812" watchObservedRunningTime="2026-07-23 17:59:14.662403143 +0000 UTC m=+104.103901737" Jul 23 17:59:14.683771 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:14.683720 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-944979c7d-2pgzv" podStartSLOduration=33.683701235 podStartE2EDuration="33.683701235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:14.682265359 +0000 UTC m=+104.123763954" watchObservedRunningTime="2026-07-23 17:59:14.683701235 +0000 UTC m=+104.125199830" Jul 23 17:59:15.012171 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.012092 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:59:15.015101 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.015079 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:59:15.051572 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.051503 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhjwd" podStartSLOduration=4.051484673 podStartE2EDuration="4.051484673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:14.702962172 +0000 UTC m=+104.144460767" watchObservedRunningTime="2026-07-23 17:59:15.051484673 +0000 UTC m=+104.492983266" Jul 23 17:59:15.178375 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.178317 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c98884d87-zhs2f"] Jul 23 17:59:15.234164 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.234114 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c98884d87-zhs2f"] Jul 23 17:59:15.234353 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.234247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.238240 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.237953 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 17:59:15.238240 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.237975 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 17:59:15.238240 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.237983 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 17:59:15.238240 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.238038 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z474v\"" Jul 23 17:59:15.238240 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.238232 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 17:59:15.238781 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.238735 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 17:59:15.238984 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.238955 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 17:59:15.239168 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.239146 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 17:59:15.395028 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.394931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgxhj\" (UniqueName: \"kubernetes.io/projected/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-kube-api-access-kgxhj\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.395028 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.394987 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-oauth-config\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.395028 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.395015 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-oauth-serving-cert\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.395301 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.395087 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-service-ca\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.395301 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.395175 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-config\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.395301 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.395219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-serving-cert\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.495825 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.495774 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-config\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.495983 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.495858 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-serving-cert\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.495983 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.495954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgxhj\" (UniqueName: \"kubernetes.io/projected/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-kube-api-access-kgxhj\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.496102 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.495985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-oauth-config\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.496102 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.496013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-oauth-serving-cert\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.496102 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.496046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-service-ca\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.496702 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.496674 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-config\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.496787 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.496724 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-service-ca\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.497934 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.497889 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-oauth-serving-cert\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.500022 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.499997 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-serving-cert\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.501037 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.501013 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-oauth-config\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.509511 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.509479 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgxhj\" (UniqueName: \"kubernetes.io/projected/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-kube-api-access-kgxhj\") pod \"console-6c98884d87-zhs2f\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.548324 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.548266 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:15.602726 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.602669 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:59:15.603061 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.603029 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-944979c7d-2pgzv" Jul 23 17:59:15.932171 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:15.932143 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c98884d87-zhs2f"] Jul 23 17:59:15.935381 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:15.935354 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e84ea4_effc_4fa6_8fbe_cc37667bb3dc.slice/crio-c7d04fa4a0a657402774096249111171ac19c425821944c50ac0ca3ed3cec323 WatchSource:0}: Error finding container c7d04fa4a0a657402774096249111171ac19c425821944c50ac0ca3ed3cec323: Status 404 returned error can't find the container with id c7d04fa4a0a657402774096249111171ac19c425821944c50ac0ca3ed3cec323 Jul 23 17:59:16.606086 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:16.606047 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c98884d87-zhs2f" event={"ID":"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc","Type":"ContainerStarted","Data":"c7d04fa4a0a657402774096249111171ac19c425821944c50ac0ca3ed3cec323"} Jul 23 17:59:17.611458 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:17.611366 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" event={"ID":"91b7eda8-056a-4c20-9923-eb26f0479ea7","Type":"ContainerStarted","Data":"a62876f9de6a4c4024c5b2972f38c7b736a96243b061a700d958b9f9b4c28207"} Jul 23 17:59:17.611897 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:17.611590 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" Jul 23 17:59:17.613278 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:17.613258 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" Jul 23 17:59:17.656000 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:17.655957 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f86564f98-49kzx" podStartSLOduration=1.987190924 podStartE2EDuration="6.655941872s" totalImagesPullingTime="4.668750948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:17.65503594 +0000 UTC m=+107.096534535" watchObservedRunningTime="2026-07-23 17:59:17.655941872 +0000 UTC m=+107.097440721" Jul 23 17:59:19.329504 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:19.329467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:59:19.332161 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:19.332131 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e46a507c-b10b-4bac-8e85-01475e644526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krgbq\" (UID: \"e46a507c-b10b-4bac-8e85-01475e644526\") " pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:59:19.577270 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:19.577226 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-krgbq" Jul 23 17:59:20.199160 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:20.199075 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-krgbq"] Jul 23 17:59:20.201958 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:20.201898 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46a507c_b10b_4bac_8e85_01475e644526.slice/crio-1de55087151396a05aeb0b536528f2eb313fbf7b5066cfb1c9f87ec3ce0e3892 WatchSource:0}: Error finding container 1de55087151396a05aeb0b536528f2eb313fbf7b5066cfb1c9f87ec3ce0e3892: Status 404 returned error can't find the container with id 1de55087151396a05aeb0b536528f2eb313fbf7b5066cfb1c9f87ec3ce0e3892 Jul 23 17:59:20.628665 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:20.628290 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c98884d87-zhs2f" event={"ID":"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc","Type":"ContainerStarted","Data":"c79d56ad96b8f11afa49ae74505debb0d1df4399bd685e32614fb329fb662b3b"} Jul 23 17:59:20.630599 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:20.630532 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krgbq" event={"ID":"e46a507c-b10b-4bac-8e85-01475e644526","Type":"ContainerStarted","Data":"1de55087151396a05aeb0b536528f2eb313fbf7b5066cfb1c9f87ec3ce0e3892"} Jul 23 17:59:20.676695 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:20.676403 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c98884d87-zhs2f" podStartSLOduration=1.520881387 podStartE2EDuration="5.676379815s" totalImagesPullingTime="4.155498428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:20.675274826 +0000 UTC m=+110.116773423" watchObservedRunningTime="2026-07-23 17:59:20.676379815 +0000 UTC m=+110.117878416" Jul 23 17:59:24.605522 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:24.605482 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5nhzx" Jul 23 17:59:25.548741 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:25.548703 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:25.548934 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:25.548771 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:25.554703 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:25.554679 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:25.653573 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:25.653531 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krgbq" event={"ID":"e46a507c-b10b-4bac-8e85-01475e644526","Type":"ContainerStarted","Data":"c3e0b4c8a7e1e7762a877ad1e9bbd5adbab6a48b00e34b6259ddd24113679183"} Jul 23 17:59:25.658743 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:25.658653 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 17:59:28.665622 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:28.665531 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krgbq" event={"ID":"e46a507c-b10b-4bac-8e85-01475e644526","Type":"ContainerStarted","Data":"dcfe7c6b231bb833798bd9d69b661df5030aae1e5f0cd87ca1c7350d8b6a4235"} Jul 23 17:59:29.160405 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.160356 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql"] Jul 23 17:59:29.165771 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.165735 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-m6zz7"] Jul 23 17:59:29.165939 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.165922 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" Jul 23 17:59:29.169359 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.169336 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.170163 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.169885 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 17:59:29.170163 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.169886 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 17:59:29.171101 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.170947 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 23 17:59:29.171101 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.170973 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-g8rhf\"" Jul 23 17:59:29.171263 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.171230 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 17:59:29.171317 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.171276 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 23 17:59:29.173947 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.173909 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8fmzl\"" Jul 23 17:59:29.174458 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.174442 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 23 17:59:29.175453 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.175436 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 23 17:59:29.176285 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.176269 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 23 17:59:29.178903 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.178833 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql"] Jul 23 17:59:29.180390 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.180369 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-m6zz7"] Jul 23 17:59:29.212309 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.212271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e4ab588-dddb-4af3-a430-a4722529b3b2-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.212485 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.212320 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b81b721b-00d4-4745-95b3-e0f93171922e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-8djql\" (UID: \"b81b721b-00d4-4745-95b3-e0f93171922e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" Jul 23 17:59:29.212485 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.212351 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6e4ab588-dddb-4af3-a430-a4722529b3b2-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.212485 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.212377 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b81b721b-00d4-4745-95b3-e0f93171922e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-8djql\" (UID: \"b81b721b-00d4-4745-95b3-e0f93171922e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" Jul 23 17:59:29.212485 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.212478 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.212696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.212503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxctv\" (UniqueName: \"kubernetes.io/projected/b81b721b-00d4-4745-95b3-e0f93171922e-kube-api-access-nxctv\") pod \"openshift-state-metrics-79f6958cdf-8djql\" (UID: \"b81b721b-00d4-4745-95b3-e0f93171922e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" Jul 23 17:59:29.212696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.212557 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b81b721b-00d4-4745-95b3-e0f93171922e-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-8djql\" (UID: \"b81b721b-00d4-4745-95b3-e0f93171922e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" Jul 23 17:59:29.212696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.212615 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.212696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.212648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pdbk\" (UniqueName: \"kubernetes.io/projected/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-api-access-9pdbk\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.212696 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.212658 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cvcp7"] Jul 23 17:59:29.212985 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.212717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.216359 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.216334 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.240234 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.240052 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 17:59:29.240234 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.240082 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c7sl6\"" Jul 23 17:59:29.240234 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.240120 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 17:59:29.240234 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.240052 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 17:59:29.313337 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313303 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d487ee73-35f1-441c-8d5d-a7b54ec44244-root\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.313531 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.313531 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313389 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdbk\" (UniqueName: \"kubernetes.io/projected/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-api-access-9pdbk\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.313531 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313426 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-tls\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.313531 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313453 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.313531 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313475 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d487ee73-35f1-441c-8d5d-a7b54ec44244-sys\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.313531 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:59:29.313543 2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e4ab588-dddb-4af3-a430-a4722529b3b2-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313590 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b81b721b-00d4-4745-95b3-e0f93171922e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-8djql\" (UID: \"b81b721b-00d4-4745-95b3-e0f93171922e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:59:29.313618 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-tls podName:6e4ab588-dddb-4af3-a430-a4722529b3b2 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:29.813596868 +0000 UTC m=+119.255095439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-m6zz7" (UID: "6e4ab588-dddb-4af3-a430-a4722529b3b2") : secret "kube-state-metrics-tls" not found Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313658 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6e4ab588-dddb-4af3-a430-a4722529b3b2-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313685 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b81b721b-00d4-4745-95b3-e0f93171922e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-8djql\" (UID: \"b81b721b-00d4-4745-95b3-e0f93171922e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313728 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4b9x\" (UniqueName: \"kubernetes.io/projected/d487ee73-35f1-441c-8d5d-a7b54ec44244-kube-api-access-f4b9x\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313784 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d487ee73-35f1-441c-8d5d-a7b54ec44244-metrics-client-ca\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313871 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxctv\" (UniqueName: \"kubernetes.io/projected/b81b721b-00d4-4745-95b3-e0f93171922e-kube-api-access-nxctv\") pod \"openshift-state-metrics-79f6958cdf-8djql\" (UID: \"b81b721b-00d4-4745-95b3-e0f93171922e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313903 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-textfile\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313929 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-accelerators-collector-config\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b81b721b-00d4-4745-95b3-e0f93171922e-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-8djql\" (UID: \"b81b721b-00d4-4745-95b3-e0f93171922e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.313991 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-wtmp\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.314850 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.314484 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6e4ab588-dddb-4af3-a430-a4722529b3b2-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.315640 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.314551 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e4ab588-dddb-4af3-a430-a4722529b3b2-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.315640 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.314678 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.315640 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.315153 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b81b721b-00d4-4745-95b3-e0f93171922e-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-8djql\" (UID: \"b81b721b-00d4-4745-95b3-e0f93171922e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" Jul 23 17:59:29.317471 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.317444 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b81b721b-00d4-4745-95b3-e0f93171922e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-8djql\" (UID: \"b81b721b-00d4-4745-95b3-e0f93171922e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" Jul 23 17:59:29.318777 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.318754 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.320102 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.320057 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b81b721b-00d4-4745-95b3-e0f93171922e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-8djql\" (UID: \"b81b721b-00d4-4745-95b3-e0f93171922e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" Jul 23 17:59:29.348738 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.348352 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxctv\" (UniqueName: \"kubernetes.io/projected/b81b721b-00d4-4745-95b3-e0f93171922e-kube-api-access-nxctv\") pod \"openshift-state-metrics-79f6958cdf-8djql\" (UID: \"b81b721b-00d4-4745-95b3-e0f93171922e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" Jul 23 17:59:29.349357 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.349007 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pdbk\" (UniqueName: \"kubernetes.io/projected/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-api-access-9pdbk\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.415440 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.414828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4b9x\" (UniqueName: \"kubernetes.io/projected/d487ee73-35f1-441c-8d5d-a7b54ec44244-kube-api-access-f4b9x\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.415440 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.414988 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d487ee73-35f1-441c-8d5d-a7b54ec44244-metrics-client-ca\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.415440 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.415023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-textfile\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.415440 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.415046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-accelerators-collector-config\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.415440 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.415079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-wtmp\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.415440 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.415111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d487ee73-35f1-441c-8d5d-a7b54ec44244-root\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.415440 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.415163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-tls\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.415440 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.415199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d487ee73-35f1-441c-8d5d-a7b54ec44244-sys\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.415440 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.415228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.415440 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.415391 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-textfile\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.415440 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.415391 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d487ee73-35f1-441c-8d5d-a7b54ec44244-root\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.416156 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:59:29.415468 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 23 17:59:29.416156 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.415476 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d487ee73-35f1-441c-8d5d-a7b54ec44244-sys\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.416156 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.415488 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-wtmp\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.416156 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:59:29.415545 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-tls podName:d487ee73-35f1-441c-8d5d-a7b54ec44244 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:29.915524246 +0000 UTC m=+119.357022835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-tls") pod "node-exporter-cvcp7" (UID: "d487ee73-35f1-441c-8d5d-a7b54ec44244") : secret "node-exporter-tls" not found Jul 23 17:59:29.416156 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.415861 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d487ee73-35f1-441c-8d5d-a7b54ec44244-metrics-client-ca\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.416433 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.416270 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-accelerators-collector-config\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.418645 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.418603 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.431994 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.431950 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4b9x\" (UniqueName: \"kubernetes.io/projected/d487ee73-35f1-441c-8d5d-a7b54ec44244-kube-api-access-f4b9x\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.482079 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.481853 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" Jul 23 17:59:29.675667 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.675202 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql"] Jul 23 17:59:29.679589 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:29.679550 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81b721b_00d4_4745_95b3_e0f93171922e.slice/crio-68866191e581eb2490227a51a05bf85e9b93c0ee7919671322fa4347ba1f031a WatchSource:0}: Error finding container 68866191e581eb2490227a51a05bf85e9b93c0ee7919671322fa4347ba1f031a: Status 404 returned error can't find the container with id 68866191e581eb2490227a51a05bf85e9b93c0ee7919671322fa4347ba1f031a Jul 23 17:59:29.820338 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.820315 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:29.820476 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:59:29.820455 2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 23 17:59:29.820546 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:59:29.820529 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-tls podName:6e4ab588-dddb-4af3-a430-a4722529b3b2 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:30.820508015 +0000 UTC m=+120.262006600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-m6zz7" (UID: "6e4ab588-dddb-4af3-a430-a4722529b3b2") : secret "kube-state-metrics-tls" not found Jul 23 17:59:29.921756 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:29.921707 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-tls\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:29.921941 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:59:29.921894 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 23 17:59:29.921994 ip-10-0-142-147 kubenswrapper[2564]: E0723 17:59:29.921973 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-tls podName:d487ee73-35f1-441c-8d5d-a7b54ec44244 nodeName:}" failed. No retries permitted until 2026-07-23 17:59:30.921952749 +0000 UTC m=+120.363451322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-tls") pod "node-exporter-cvcp7" (UID: "d487ee73-35f1-441c-8d5d-a7b54ec44244") : secret "node-exporter-tls" not found Jul 23 17:59:30.677422 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:30.677389 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" event={"ID":"b81b721b-00d4-4745-95b3-e0f93171922e","Type":"ContainerStarted","Data":"653cb2cb7821101788129dbc90cab45b884345836bc1283097b28dafa46ae431"} Jul 23 17:59:30.677860 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:30.677456 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" event={"ID":"b81b721b-00d4-4745-95b3-e0f93171922e","Type":"ContainerStarted","Data":"24defa152fd5d51cce2ae82512d9d0aa522502069b1c8eb481756a4d0d002cca"} Jul 23 17:59:30.677860 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:30.677483 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" event={"ID":"b81b721b-00d4-4745-95b3-e0f93171922e","Type":"ContainerStarted","Data":"68866191e581eb2490227a51a05bf85e9b93c0ee7919671322fa4347ba1f031a"} Jul 23 17:59:30.828496 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:30.828454 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:30.831193 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:30.831167 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e4ab588-dddb-4af3-a430-a4722529b3b2-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-m6zz7\" (UID: \"6e4ab588-dddb-4af3-a430-a4722529b3b2\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:30.929952 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:30.929867 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-tls\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:30.932326 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:30.932292 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d487ee73-35f1-441c-8d5d-a7b54ec44244-node-exporter-tls\") pod \"node-exporter-cvcp7\" (UID: \"d487ee73-35f1-441c-8d5d-a7b54ec44244\") " pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:30.993417 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:30.993389 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8fmzl\"" Jul 23 17:59:30.999968 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:30.999940 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" Jul 23 17:59:31.031126 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:31.031095 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c7sl6\"" Jul 23 17:59:31.038310 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:31.038286 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cvcp7" Jul 23 17:59:32.287152 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.287116 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-79644dc894-vbcpf"] Jul 23 17:59:32.291079 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.291055 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.294217 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.294190 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 23 17:59:32.294341 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.294250 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xqx56\"" Jul 23 17:59:32.295083 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.294995 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 23 17:59:32.295083 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.295017 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e7epgcb1pfmuc\"" Jul 23 17:59:32.295671 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.295649 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 23 17:59:32.296036 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.296018 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 23 17:59:32.301563 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.301546 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 23 17:59:32.312674 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.312651 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79644dc894-vbcpf"] Jul 23 17:59:32.343139 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.343108 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.343290 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.343176 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-tls\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.343343 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.343297 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a13fca3-602d-41b5-8057-fc0a786f13aa-metrics-client-ca\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.343343 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.343331 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.343436 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.343392 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.343482 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.343431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-grpc-tls\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.343519 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.343502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.343559 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.343535 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbr54\" (UniqueName: \"kubernetes.io/projected/6a13fca3-602d-41b5-8057-fc0a786f13aa-kube-api-access-vbr54\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.445612 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.445093 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.445612 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.445168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-grpc-tls\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.445612 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.445244 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.445612 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.445299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbr54\" (UniqueName: \"kubernetes.io/projected/6a13fca3-602d-41b5-8057-fc0a786f13aa-kube-api-access-vbr54\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.445612 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.445343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.445612 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.445407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-tls\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.445612 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.445532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a13fca3-602d-41b5-8057-fc0a786f13aa-metrics-client-ca\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.445612 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.445574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.447417 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.446502 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a13fca3-602d-41b5-8057-fc0a786f13aa-metrics-client-ca\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.448411 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:32.448380 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd487ee73_35f1_441c_8d5d_a7b54ec44244.slice/crio-820e681e015826da102f1c5595bb368c4a981a8790bc7bcb71040b71e5ab9d9b WatchSource:0}: Error finding container 820e681e015826da102f1c5595bb368c4a981a8790bc7bcb71040b71e5ab9d9b: Status 404 returned error can't find the container with id 820e681e015826da102f1c5595bb368c4a981a8790bc7bcb71040b71e5ab9d9b Jul 23 17:59:32.449893 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.449847 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.451581 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.451510 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-grpc-tls\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.452762 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.452732 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.454291 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.452941 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.454291 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.453609 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.454291 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.453732 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6a13fca3-602d-41b5-8057-fc0a786f13aa-secret-thanos-querier-tls\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.467449 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.467426 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbr54\" (UniqueName: \"kubernetes.io/projected/6a13fca3-602d-41b5-8057-fc0a786f13aa-kube-api-access-vbr54\") pod \"thanos-querier-79644dc894-vbcpf\" (UID: \"6a13fca3-602d-41b5-8057-fc0a786f13aa\") " pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.602314 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.602236 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:32.622008 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.621979 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-m6zz7"] Jul 23 17:59:32.624931 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:32.624884 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4ab588_dddb_4af3_a430_a4722529b3b2.slice/crio-c138779d285d17db68589ffb5dd33c3e35bf4d5feb3fee85180e6a9739740360 WatchSource:0}: Error finding container c138779d285d17db68589ffb5dd33c3e35bf4d5feb3fee85180e6a9739740360: Status 404 returned error can't find the container with id c138779d285d17db68589ffb5dd33c3e35bf4d5feb3fee85180e6a9739740360 Jul 23 17:59:32.695872 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.695816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" event={"ID":"b81b721b-00d4-4745-95b3-e0f93171922e","Type":"ContainerStarted","Data":"2d6640a4c75b755056cef60a6898062772f1cd8b76b7a8f59732f510f18f5a39"} Jul 23 17:59:32.697328 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.697299 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cvcp7" event={"ID":"d487ee73-35f1-441c-8d5d-a7b54ec44244","Type":"ContainerStarted","Data":"820e681e015826da102f1c5595bb368c4a981a8790bc7bcb71040b71e5ab9d9b"} Jul 23 17:59:32.698818 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.698773 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" event={"ID":"6e4ab588-dddb-4af3-a430-a4722529b3b2","Type":"ContainerStarted","Data":"c138779d285d17db68589ffb5dd33c3e35bf4d5feb3fee85180e6a9739740360"} Jul 23 17:59:32.701432 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.701407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krgbq" event={"ID":"e46a507c-b10b-4bac-8e85-01475e644526","Type":"ContainerStarted","Data":"e80d18bf48f723dabfeecb72108dfa9b63a3bb687ca14d079ddf99fa47b1c9c0"} Jul 23 17:59:32.733324 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.733267 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8djql" podStartSLOduration=1.115855939 podStartE2EDuration="3.733246697s" totalImagesPullingTime="2.617390758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:32.732524689 +0000 UTC m=+122.174023284" watchObservedRunningTime="2026-07-23 17:59:32.733246697 +0000 UTC m=+122.174745293" Jul 23 17:59:32.772558 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.772495 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-krgbq" podStartSLOduration=33.626750824 podStartE2EDuration="45.772475617s" totalImagesPullingTime="12.145724793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:58:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:32.772439118 +0000 UTC m=+122.213937714" watchObservedRunningTime="2026-07-23 17:59:32.772475617 +0000 UTC m=+122.213974207" Jul 23 17:59:32.792044 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:32.791970 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79644dc894-vbcpf"] Jul 23 17:59:32.795067 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:32.795033 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a13fca3_602d_41b5_8057_fc0a786f13aa.slice/crio-4d11af0d74176edac7d0a6df3fe08b86282ae56aff19423fdcc5cd0a08e95626 WatchSource:0}: Error finding container 4d11af0d74176edac7d0a6df3fe08b86282ae56aff19423fdcc5cd0a08e95626: Status 404 returned error can't find the container with id 4d11af0d74176edac7d0a6df3fe08b86282ae56aff19423fdcc5cd0a08e95626 Jul 23 17:59:33.708432 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:33.708337 2564 generic.go:356] "Generic (PLEG): container finished" podID="d487ee73-35f1-441c-8d5d-a7b54ec44244" containerID="5648c8f347768b3cfb05caa7f1740826b2773f074e8b0a25a0c6fe62c490351f" exitCode=0 Jul 23 17:59:33.708894 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:33.708596 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cvcp7" event={"ID":"d487ee73-35f1-441c-8d5d-a7b54ec44244","Type":"ContainerDied","Data":"5648c8f347768b3cfb05caa7f1740826b2773f074e8b0a25a0c6fe62c490351f"} Jul 23 17:59:33.710197 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:33.710096 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" event={"ID":"6a13fca3-602d-41b5-8057-fc0a786f13aa","Type":"ContainerStarted","Data":"4d11af0d74176edac7d0a6df3fe08b86282ae56aff19423fdcc5cd0a08e95626"} Jul 23 17:59:33.936053 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:33.936014 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs"] Jul 23 17:59:33.939869 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:33.939847 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs" Jul 23 17:59:33.943753 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:33.943725 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4srkw\"" Jul 23 17:59:33.949372 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:33.949352 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 23 17:59:33.962697 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:33.962624 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs"] Jul 23 17:59:34.061975 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:34.061938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e467ddca-d02c-4880-a006-9190caca249a-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-pgnbs\" (UID: \"e467ddca-d02c-4880-a006-9190caca249a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs" Jul 23 17:59:34.062142 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:34.062062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxtkx\" (UniqueName: \"kubernetes.io/projected/e467ddca-d02c-4880-a006-9190caca249a-kube-api-access-vxtkx\") pod \"monitoring-plugin-7b7b7bc8bf-pgnbs\" (UID: \"e467ddca-d02c-4880-a006-9190caca249a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs" Jul 23 17:59:34.163060 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:34.163023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e467ddca-d02c-4880-a006-9190caca249a-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-pgnbs\" (UID: \"e467ddca-d02c-4880-a006-9190caca249a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs" Jul 23 17:59:34.163224 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:34.163109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxtkx\" (UniqueName: \"kubernetes.io/projected/e467ddca-d02c-4880-a006-9190caca249a-kube-api-access-vxtkx\") pod \"monitoring-plugin-7b7b7bc8bf-pgnbs\" (UID: \"e467ddca-d02c-4880-a006-9190caca249a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs" Jul 23 17:59:34.165695 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:34.165669 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e467ddca-d02c-4880-a006-9190caca249a-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-pgnbs\" (UID: \"e467ddca-d02c-4880-a006-9190caca249a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs" Jul 23 17:59:34.177491 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:34.177459 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxtkx\" (UniqueName: \"kubernetes.io/projected/e467ddca-d02c-4880-a006-9190caca249a-kube-api-access-vxtkx\") pod \"monitoring-plugin-7b7b7bc8bf-pgnbs\" (UID: \"e467ddca-d02c-4880-a006-9190caca249a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs" Jul 23 17:59:34.252357 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:34.252323 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs" Jul 23 17:59:34.717408 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:34.717372 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cvcp7" event={"ID":"d487ee73-35f1-441c-8d5d-a7b54ec44244","Type":"ContainerStarted","Data":"27bb26ff8b99337c906c583906c1dd058c8ee35b684564fb6769a7d47b7c5c74"} Jul 23 17:59:34.717854 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:34.717432 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cvcp7" event={"ID":"d487ee73-35f1-441c-8d5d-a7b54ec44244","Type":"ContainerStarted","Data":"bae465d3057b18c75db3294cadc52d0fc0553d7c22ab09546cb455dda7f4dd92"} Jul 23 17:59:34.752408 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:34.752352 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cvcp7" podStartSLOduration=4.899091425 podStartE2EDuration="5.75232805s" totalImagesPullingTime="785.848039ms" totalInitContainerRuntime="67.388586ms" isStatefulPod=true podCreationTimestamp="2026-07-23 17:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:34.75100932 +0000 UTC m=+124.192507916" watchObservedRunningTime="2026-07-23 17:59:34.75232805 +0000 UTC m=+124.193826645" Jul 23 17:59:34.943431 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:34.943405 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs"] Jul 23 17:59:34.948659 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:34.948632 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode467ddca_d02c_4880_a006_9190caca249a.slice/crio-5857d69d2960b5fa3cdb4d187e4c2d1e47584c9d4cb75fdfe2374ec59480ae41 WatchSource:0}: Error finding container 5857d69d2960b5fa3cdb4d187e4c2d1e47584c9d4cb75fdfe2374ec59480ae41: Status 404 returned error can't find the container with id 5857d69d2960b5fa3cdb4d187e4c2d1e47584c9d4cb75fdfe2374ec59480ae41 Jul 23 17:59:35.607742 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:35.607712 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-db4467ff9-p7t8v" Jul 23 17:59:35.726712 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:35.726675 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" event={"ID":"6a13fca3-602d-41b5-8057-fc0a786f13aa","Type":"ContainerStarted","Data":"9c808c5e788acfa5fbcdde37f68f14a38322f9c60b345a8b7e6daed5f086ce7b"} Jul 23 17:59:35.727187 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:35.726758 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" event={"ID":"6a13fca3-602d-41b5-8057-fc0a786f13aa","Type":"ContainerStarted","Data":"df9fb615b226608d3632f26fc32bab40fdddf857a7c779eccfc3f96a72ddbe65"} Jul 23 17:59:35.727187 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:35.726819 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" event={"ID":"6a13fca3-602d-41b5-8057-fc0a786f13aa","Type":"ContainerStarted","Data":"eb0c16dcf71e06493605b93de198600bb937d972a024d239dfe58fd5fac7a006"} Jul 23 17:59:35.728170 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:35.728141 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs" event={"ID":"e467ddca-d02c-4880-a006-9190caca249a","Type":"ContainerStarted","Data":"5857d69d2960b5fa3cdb4d187e4c2d1e47584c9d4cb75fdfe2374ec59480ae41"} Jul 23 17:59:35.730645 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:35.730610 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" event={"ID":"6e4ab588-dddb-4af3-a430-a4722529b3b2","Type":"ContainerStarted","Data":"d639fa8ab8f38fb5ba1a0dc3d6545767a09a6641978fe830e73a4009fa25f170"} Jul 23 17:59:35.730753 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:35.730668 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" event={"ID":"6e4ab588-dddb-4af3-a430-a4722529b3b2","Type":"ContainerStarted","Data":"00f956f92d7b8c02b585cf71ce08b844bb43e6d06874d498f1f73d8c5d8ef20f"} Jul 23 17:59:35.730753 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:35.730690 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" event={"ID":"6e4ab588-dddb-4af3-a430-a4722529b3b2","Type":"ContainerStarted","Data":"65efb57a8305a6c9941fd36752a6f7761e6b953ae331acc7614859494d5031b1"} Jul 23 17:59:36.737415 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:36.737379 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" event={"ID":"6a13fca3-602d-41b5-8057-fc0a786f13aa","Type":"ContainerStarted","Data":"d770a1e787d9d498e4b9a882a1c804e30accbbbc19bec1b3697e36a660530852"} Jul 23 17:59:36.737915 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:36.737458 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" event={"ID":"6a13fca3-602d-41b5-8057-fc0a786f13aa","Type":"ContainerStarted","Data":"100ba820ff88d6ea19c4ed35fb22f438fa767e97c689f264b02d157a606b0f66"} Jul 23 17:59:36.737915 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:36.737490 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" event={"ID":"6a13fca3-602d-41b5-8057-fc0a786f13aa","Type":"ContainerStarted","Data":"f5e902521ea9e7495279c8e7640b2145f74e421bfafafe115d4a56fbf9fc28b1"} Jul 23 17:59:36.737915 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:36.737529 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:36.738960 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:36.738940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs" event={"ID":"e467ddca-d02c-4880-a006-9190caca249a","Type":"ContainerStarted","Data":"2eb438be199ec22da5be14705b8a986856f9d5b064200df5801dd3903b50f3c1"} Jul 23 17:59:36.739326 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:36.739308 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs" Jul 23 17:59:36.744643 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:36.744620 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs" Jul 23 17:59:36.808467 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:36.808386 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" podStartSLOduration=1.45050823 podStartE2EDuration="4.808365615s" totalImagesPullingTime="3.357857385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:36.805250535 +0000 UTC m=+126.246749153" watchObservedRunningTime="2026-07-23 17:59:36.808365615 +0000 UTC m=+126.249864212" Jul 23 17:59:36.808994 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:36.808956 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-m6zz7" podStartSLOduration=5.646290237 podStartE2EDuration="7.808943914s" totalImagesPullingTime="2.162653677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:36.044953008 +0000 UTC m=+125.486451602" watchObservedRunningTime="2026-07-23 17:59:36.808943914 +0000 UTC m=+126.250442509" Jul 23 17:59:36.839798 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:36.839746 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-pgnbs" podStartSLOduration=2.400083038 podStartE2EDuration="3.839729079s" totalImagesPullingTime="1.439646041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:36.839542096 +0000 UTC m=+126.281040712" watchObservedRunningTime="2026-07-23 17:59:36.839729079 +0000 UTC m=+126.281227663" Jul 23 17:59:37.042542 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.042460 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54bcccd55-pztvt"] Jul 23 17:59:37.045796 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.045775 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.068344 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.068313 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 17:59:37.092166 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.092132 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-oauth-config\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.092308 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.092203 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-serving-cert\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.092308 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.092225 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-oauth-serving-cert\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.092308 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.092298 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54bcccd55-pztvt"] Jul 23 17:59:37.092516 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.092318 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-config\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.092516 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.092350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dx8w\" (UniqueName: \"kubernetes.io/projected/9ac336c7-36ca-40a4-942e-7f11cb4590bb-kube-api-access-5dx8w\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.092516 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.092389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-trusted-ca-bundle\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.092516 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.092443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-service-ca\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.193744 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.193710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dx8w\" (UniqueName: \"kubernetes.io/projected/9ac336c7-36ca-40a4-942e-7f11cb4590bb-kube-api-access-5dx8w\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.193744 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.193748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-trusted-ca-bundle\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.193965 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.193773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-service-ca\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.193965 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.193823 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-oauth-config\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.193965 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.193891 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-serving-cert\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.193965 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.193918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-oauth-serving-cert\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.194128 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.193996 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-config\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.194557 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.194531 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-service-ca\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.194682 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.194655 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-oauth-serving-cert\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.194733 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.194698 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-config\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.194765 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.194726 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-trusted-ca-bundle\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.196252 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.196229 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-serving-cert\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.196339 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.196264 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-oauth-config\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.207012 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.206990 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dx8w\" (UniqueName: \"kubernetes.io/projected/9ac336c7-36ca-40a4-942e-7f11cb4590bb-kube-api-access-5dx8w\") pod \"console-54bcccd55-pztvt\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.354403 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.354321 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:37.504727 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.504653 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54bcccd55-pztvt"] Jul 23 17:59:37.507079 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:37.507055 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac336c7_36ca_40a4_942e_7f11cb4590bb.slice/crio-e4920da860346e6719898775aa4354a4004dff31acd72ed858e7c349a27d315c WatchSource:0}: Error finding container e4920da860346e6719898775aa4354a4004dff31acd72ed858e7c349a27d315c: Status 404 returned error can't find the container with id e4920da860346e6719898775aa4354a4004dff31acd72ed858e7c349a27d315c Jul 23 17:59:37.744709 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.744676 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54bcccd55-pztvt" event={"ID":"9ac336c7-36ca-40a4-942e-7f11cb4590bb","Type":"ContainerStarted","Data":"9400cef12118934cb6fb042588e766f1b20e37c255898ad525e1fcc7724c6db3"} Jul 23 17:59:37.745158 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.744734 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54bcccd55-pztvt" event={"ID":"9ac336c7-36ca-40a4-942e-7f11cb4590bb","Type":"ContainerStarted","Data":"e4920da860346e6719898775aa4354a4004dff31acd72ed858e7c349a27d315c"} Jul 23 17:59:37.778057 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:37.778008 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54bcccd55-pztvt" podStartSLOduration=1.7779924820000002 podStartE2EDuration="1.777992482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:37.777713741 +0000 UTC m=+127.219212334" watchObservedRunningTime="2026-07-23 17:59:37.777992482 +0000 UTC m=+127.219491075" Jul 23 17:59:39.817710 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:39.817662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs\") pod \"network-metrics-daemon-lg9sf\" (UID: \"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26\") " pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:59:39.820071 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:39.820040 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26-metrics-certs\") pod \"network-metrics-daemon-lg9sf\" (UID: \"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26\") " pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:59:39.925822 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:39.925786 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7wdhp\"" Jul 23 17:59:39.933465 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:39.933450 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg9sf" Jul 23 17:59:40.080451 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:40.080364 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lg9sf"] Jul 23 17:59:40.083277 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:40.083247 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5aa935f_c84e_4eb8_a2c6_09cbe3f54b26.slice/crio-0602ab11da0c31e5adc525e2fa312c9e90a9db247b918704eb6a6753c1954ba6 WatchSource:0}: Error finding container 0602ab11da0c31e5adc525e2fa312c9e90a9db247b918704eb6a6753c1954ba6: Status 404 returned error can't find the container with id 0602ab11da0c31e5adc525e2fa312c9e90a9db247b918704eb6a6753c1954ba6 Jul 23 17:59:40.757528 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:40.757480 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lg9sf" event={"ID":"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26","Type":"ContainerStarted","Data":"0602ab11da0c31e5adc525e2fa312c9e90a9db247b918704eb6a6753c1954ba6"} Jul 23 17:59:41.762753 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:41.762721 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lg9sf" event={"ID":"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26","Type":"ContainerStarted","Data":"6eddd86c149b4a889c385acacf8dbed861ff9fc01305db25ef6f2a88a8313e0a"} Jul 23 17:59:41.763122 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:41.762776 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lg9sf" event={"ID":"b5aa935f-c84e-4eb8-a2c6-09cbe3f54b26","Type":"ContainerStarted","Data":"82ebfa3cc055801f98137ce3c04c83c3140ca5196205b1a30c078ea3a291ce6b"} Jul 23 17:59:42.750288 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:42.750260 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-79644dc894-vbcpf" Jul 23 17:59:42.786713 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:42.786652 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lg9sf" podStartSLOduration=130.900314068 podStartE2EDuration="2m11.786635275s" totalImagesPullingTime="886.321207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:41.954988267 +0000 UTC m=+131.396486863" watchObservedRunningTime="2026-07-23 17:59:42.786635275 +0000 UTC m=+132.228133868" Jul 23 17:59:43.851931 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:43.851897 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54bcccd55-pztvt"] Jul 23 17:59:43.961627 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:43.961588 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d8d9f6f9d-w7qxx"] Jul 23 17:59:43.965730 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:43.965710 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:43.991679 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:43.991652 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d8d9f6f9d-w7qxx"] Jul 23 17:59:44.057423 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.057389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62c6t\" (UniqueName: \"kubernetes.io/projected/88e2eae4-dbfb-4955-8f79-6efd44616f8a-kube-api-access-62c6t\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.057583 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.057454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-serving-cert\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.057583 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.057481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-trusted-ca-bundle\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.057583 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.057504 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-service-ca\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.057583 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.057571 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-oauth-config\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.057782 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.057598 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-config\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.057782 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.057691 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-oauth-serving-cert\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.158849 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.158738 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-oauth-serving-cert\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.158849 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.158784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62c6t\" (UniqueName: \"kubernetes.io/projected/88e2eae4-dbfb-4955-8f79-6efd44616f8a-kube-api-access-62c6t\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.159038 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.158923 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-serving-cert\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.159038 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.158952 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-trusted-ca-bundle\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.159038 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.158980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-service-ca\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.159148 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.159110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-oauth-config\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.159198 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.159152 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-config\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.159560 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.159534 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-oauth-serving-cert\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.159786 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.159762 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-service-ca\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.159937 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.159915 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-config\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.160079 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.160061 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-trusted-ca-bundle\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.161489 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.161466 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-oauth-config\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.161615 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.161594 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-serving-cert\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.170287 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.170256 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62c6t\" (UniqueName: \"kubernetes.io/projected/88e2eae4-dbfb-4955-8f79-6efd44616f8a-kube-api-access-62c6t\") pod \"console-d8d9f6f9d-w7qxx\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.274429 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.274392 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:44.428581 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.428501 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d8d9f6f9d-w7qxx"] Jul 23 17:59:44.431220 ip-10-0-142-147 kubenswrapper[2564]: W0723 17:59:44.431194 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e2eae4_dbfb_4955_8f79_6efd44616f8a.slice/crio-7bea4186a367a8ef387afcd06ac23402028fd0981092648a2a9d810feb1c3a59 WatchSource:0}: Error finding container 7bea4186a367a8ef387afcd06ac23402028fd0981092648a2a9d810feb1c3a59: Status 404 returned error can't find the container with id 7bea4186a367a8ef387afcd06ac23402028fd0981092648a2a9d810feb1c3a59 Jul 23 17:59:44.775196 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.775160 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d8d9f6f9d-w7qxx" event={"ID":"88e2eae4-dbfb-4955-8f79-6efd44616f8a","Type":"ContainerStarted","Data":"0aacf7272221deaaa52c79e107c73e9e0125b13e993879d1757b1979313b5d13"} Jul 23 17:59:44.775381 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.775209 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d8d9f6f9d-w7qxx" event={"ID":"88e2eae4-dbfb-4955-8f79-6efd44616f8a","Type":"ContainerStarted","Data":"7bea4186a367a8ef387afcd06ac23402028fd0981092648a2a9d810feb1c3a59"} Jul 23 17:59:44.813735 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:44.813675 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d8d9f6f9d-w7qxx" podStartSLOduration=1.8136578110000001 podStartE2EDuration="1.813657811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:59:44.811041769 +0000 UTC m=+134.252540375" watchObservedRunningTime="2026-07-23 17:59:44.813657811 +0000 UTC m=+134.255156404" Jul 23 17:59:47.354580 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:47.354545 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54bcccd55-pztvt" Jul 23 17:59:54.275446 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:54.275405 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:54.275877 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:54.275472 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:54.280285 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:54.280263 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:54.817935 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:54.817902 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 17:59:54.892456 ip-10-0-142-147 kubenswrapper[2564]: I0723 17:59:54.892426 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c98884d87-zhs2f"] Jul 23 18:00:00.838586 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:00.838550 2564 generic.go:356] "Generic (PLEG): container finished" podID="97ce2dab-10a1-4fa6-818d-ec1aa4586798" containerID="2f109f6f374511b94f5118e29315013dee796c5bfde50cdcb714bffa278ae5e4" exitCode=0 Jul 23 18:00:00.838984 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:00.838606 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" event={"ID":"97ce2dab-10a1-4fa6-818d-ec1aa4586798","Type":"ContainerDied","Data":"2f109f6f374511b94f5118e29315013dee796c5bfde50cdcb714bffa278ae5e4"} Jul 23 18:00:00.838984 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:00.838959 2564 scope.go:117] "RemoveContainer" containerID="2f109f6f374511b94f5118e29315013dee796c5bfde50cdcb714bffa278ae5e4" Jul 23 18:00:01.844474 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:01.844436 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-8bqz6" event={"ID":"97ce2dab-10a1-4fa6-818d-ec1aa4586798","Type":"ContainerStarted","Data":"6d379aef2ef05a65327d5e63ddb903e9c5e9ce10a50c8411e038ae39fef7f83e"} Jul 23 18:00:08.871530 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:08.871456 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54bcccd55-pztvt" podUID="9ac336c7-36ca-40a4-942e-7f11cb4590bb" containerName="console" containerID="cri-o://9400cef12118934cb6fb042588e766f1b20e37c255898ad525e1fcc7724c6db3" gracePeriod=15 Jul 23 18:00:09.131339 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.131285 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54bcccd55-pztvt" Jul 23 18:00:09.175223 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.175192 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-serving-cert\") pod \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " Jul 23 18:00:09.175410 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.175234 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dx8w\" (UniqueName: \"kubernetes.io/projected/9ac336c7-36ca-40a4-942e-7f11cb4590bb-kube-api-access-5dx8w\") pod \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " Jul 23 18:00:09.175410 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.175275 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-service-ca\") pod \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " Jul 23 18:00:09.175410 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.175320 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-oauth-config\") pod \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " Jul 23 18:00:09.175410 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.175338 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-config\") pod \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " Jul 23 18:00:09.175818 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.175738 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-service-ca" (OuterVolumeSpecName: "service-ca") pod "9ac336c7-36ca-40a4-942e-7f11cb4590bb" (UID: "9ac336c7-36ca-40a4-942e-7f11cb4590bb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:00:09.175818 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.175772 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-config" (OuterVolumeSpecName: "console-config") pod "9ac336c7-36ca-40a4-942e-7f11cb4590bb" (UID: "9ac336c7-36ca-40a4-942e-7f11cb4590bb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:00:09.176017 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.175823 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-oauth-serving-cert\") pod \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " Jul 23 18:00:09.176017 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.175859 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-trusted-ca-bundle\") pod \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\" (UID: \"9ac336c7-36ca-40a4-942e-7f11cb4590bb\") " Jul 23 18:00:09.176245 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.176159 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-config\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:00:09.176245 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.176166 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9ac336c7-36ca-40a4-942e-7f11cb4590bb" (UID: "9ac336c7-36ca-40a4-942e-7f11cb4590bb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:00:09.176245 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.176178 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-service-ca\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:00:09.176500 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.176439 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9ac336c7-36ca-40a4-942e-7f11cb4590bb" (UID: "9ac336c7-36ca-40a4-942e-7f11cb4590bb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:00:09.178124 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.178078 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9ac336c7-36ca-40a4-942e-7f11cb4590bb" (UID: "9ac336c7-36ca-40a4-942e-7f11cb4590bb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:00:09.178124 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.178100 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9ac336c7-36ca-40a4-942e-7f11cb4590bb" (UID: "9ac336c7-36ca-40a4-942e-7f11cb4590bb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:00:09.178292 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.178267 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac336c7-36ca-40a4-942e-7f11cb4590bb-kube-api-access-5dx8w" (OuterVolumeSpecName: "kube-api-access-5dx8w") pod "9ac336c7-36ca-40a4-942e-7f11cb4590bb" (UID: "9ac336c7-36ca-40a4-942e-7f11cb4590bb"). InnerVolumeSpecName "kube-api-access-5dx8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:00:09.277197 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.277158 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-oauth-serving-cert\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:00:09.277197 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.277189 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ac336c7-36ca-40a4-942e-7f11cb4590bb-trusted-ca-bundle\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:00:09.277197 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.277200 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-serving-cert\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:00:09.277430 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.277208 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dx8w\" (UniqueName: \"kubernetes.io/projected/9ac336c7-36ca-40a4-942e-7f11cb4590bb-kube-api-access-5dx8w\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:00:09.277430 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.277217 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ac336c7-36ca-40a4-942e-7f11cb4590bb-console-oauth-config\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:00:09.885247 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.885214 2564 generic.go:356] "Generic (PLEG): container finished" podID="9ac336c7-36ca-40a4-942e-7f11cb4590bb" containerID="9400cef12118934cb6fb042588e766f1b20e37c255898ad525e1fcc7724c6db3" exitCode=0 Jul 23 18:00:09.885622 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.885268 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54bcccd55-pztvt" Jul 23 18:00:09.885622 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.885294 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54bcccd55-pztvt" event={"ID":"9ac336c7-36ca-40a4-942e-7f11cb4590bb","Type":"ContainerDied","Data":"9400cef12118934cb6fb042588e766f1b20e37c255898ad525e1fcc7724c6db3"} Jul 23 18:00:09.885622 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.885386 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54bcccd55-pztvt" event={"ID":"9ac336c7-36ca-40a4-942e-7f11cb4590bb","Type":"ContainerDied","Data":"e4920da860346e6719898775aa4354a4004dff31acd72ed858e7c349a27d315c"} Jul 23 18:00:09.885622 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.885411 2564 scope.go:117] "RemoveContainer" containerID="9400cef12118934cb6fb042588e766f1b20e37c255898ad525e1fcc7724c6db3" Jul 23 18:00:09.894629 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.894600 2564 scope.go:117] "RemoveContainer" containerID="9400cef12118934cb6fb042588e766f1b20e37c255898ad525e1fcc7724c6db3" Jul 23 18:00:09.894985 ip-10-0-142-147 kubenswrapper[2564]: E0723 18:00:09.894954 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9400cef12118934cb6fb042588e766f1b20e37c255898ad525e1fcc7724c6db3\": container with ID starting with 9400cef12118934cb6fb042588e766f1b20e37c255898ad525e1fcc7724c6db3 not found: ID does not exist" containerID="9400cef12118934cb6fb042588e766f1b20e37c255898ad525e1fcc7724c6db3" Jul 23 18:00:09.895096 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.894997 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9400cef12118934cb6fb042588e766f1b20e37c255898ad525e1fcc7724c6db3"} err="failed to get container status \"9400cef12118934cb6fb042588e766f1b20e37c255898ad525e1fcc7724c6db3\": rpc error: code = NotFound desc = could not find container \"9400cef12118934cb6fb042588e766f1b20e37c255898ad525e1fcc7724c6db3\": container with ID starting with 9400cef12118934cb6fb042588e766f1b20e37c255898ad525e1fcc7724c6db3 not found: ID does not exist" Jul 23 18:00:09.909048 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.908996 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54bcccd55-pztvt"] Jul 23 18:00:09.913519 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:09.913491 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54bcccd55-pztvt"] Jul 23 18:00:11.116264 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:11.116226 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ac336c7-36ca-40a4-942e-7f11cb4590bb" path="/var/lib/kubelet/pods/9ac336c7-36ca-40a4-942e-7f11cb4590bb/volumes" Jul 23 18:00:19.914399 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:19.914331 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c98884d87-zhs2f" podUID="84e84ea4-effc-4fa6-8fbe-cc37667bb3dc" containerName="console" containerID="cri-o://c79d56ad96b8f11afa49ae74505debb0d1df4399bd685e32614fb329fb662b3b" gracePeriod=15 Jul 23 18:00:20.159542 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.159516 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 18:00:20.268887 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.268848 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-oauth-config\") pod \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " Jul 23 18:00:20.269091 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.268908 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-oauth-serving-cert\") pod \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " Jul 23 18:00:20.269091 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.268958 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-serving-cert\") pod \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " Jul 23 18:00:20.269091 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.268984 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgxhj\" (UniqueName: \"kubernetes.io/projected/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-kube-api-access-kgxhj\") pod \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " Jul 23 18:00:20.269091 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.269014 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-config\") pod \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " Jul 23 18:00:20.269091 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.269069 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-service-ca\") pod \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\" (UID: \"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc\") " Jul 23 18:00:20.269532 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.269503 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "84e84ea4-effc-4fa6-8fbe-cc37667bb3dc" (UID: "84e84ea4-effc-4fa6-8fbe-cc37667bb3dc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:00:20.269655 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.269551 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-config" (OuterVolumeSpecName: "console-config") pod "84e84ea4-effc-4fa6-8fbe-cc37667bb3dc" (UID: "84e84ea4-effc-4fa6-8fbe-cc37667bb3dc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:00:20.269655 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.269595 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-service-ca" (OuterVolumeSpecName: "service-ca") pod "84e84ea4-effc-4fa6-8fbe-cc37667bb3dc" (UID: "84e84ea4-effc-4fa6-8fbe-cc37667bb3dc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:00:20.271115 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.271087 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "84e84ea4-effc-4fa6-8fbe-cc37667bb3dc" (UID: "84e84ea4-effc-4fa6-8fbe-cc37667bb3dc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:00:20.271190 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.271154 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "84e84ea4-effc-4fa6-8fbe-cc37667bb3dc" (UID: "84e84ea4-effc-4fa6-8fbe-cc37667bb3dc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:00:20.271243 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.271225 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-kube-api-access-kgxhj" (OuterVolumeSpecName: "kube-api-access-kgxhj") pod "84e84ea4-effc-4fa6-8fbe-cc37667bb3dc" (UID: "84e84ea4-effc-4fa6-8fbe-cc37667bb3dc"). InnerVolumeSpecName "kube-api-access-kgxhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:00:20.370123 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.370075 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-oauth-serving-cert\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:00:20.370123 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.370115 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-serving-cert\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:00:20.370123 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.370125 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgxhj\" (UniqueName: \"kubernetes.io/projected/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-kube-api-access-kgxhj\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:00:20.370123 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.370136 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-config\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:00:20.370382 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.370145 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-service-ca\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:00:20.370382 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.370152 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc-console-oauth-config\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:00:20.930527 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.930490 2564 generic.go:356] "Generic (PLEG): container finished" podID="84e84ea4-effc-4fa6-8fbe-cc37667bb3dc" containerID="c79d56ad96b8f11afa49ae74505debb0d1df4399bd685e32614fb329fb662b3b" exitCode=0 Jul 23 18:00:20.930980 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.930538 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c98884d87-zhs2f" Jul 23 18:00:20.930980 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.930575 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c98884d87-zhs2f" event={"ID":"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc","Type":"ContainerDied","Data":"c79d56ad96b8f11afa49ae74505debb0d1df4399bd685e32614fb329fb662b3b"} Jul 23 18:00:20.930980 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.930630 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c98884d87-zhs2f" event={"ID":"84e84ea4-effc-4fa6-8fbe-cc37667bb3dc","Type":"ContainerDied","Data":"c7d04fa4a0a657402774096249111171ac19c425821944c50ac0ca3ed3cec323"} Jul 23 18:00:20.930980 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.930655 2564 scope.go:117] "RemoveContainer" containerID="c79d56ad96b8f11afa49ae74505debb0d1df4399bd685e32614fb329fb662b3b" Jul 23 18:00:20.938938 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.938919 2564 scope.go:117] "RemoveContainer" containerID="c79d56ad96b8f11afa49ae74505debb0d1df4399bd685e32614fb329fb662b3b" Jul 23 18:00:20.939173 ip-10-0-142-147 kubenswrapper[2564]: E0723 18:00:20.939154 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79d56ad96b8f11afa49ae74505debb0d1df4399bd685e32614fb329fb662b3b\": container with ID starting with c79d56ad96b8f11afa49ae74505debb0d1df4399bd685e32614fb329fb662b3b not found: ID does not exist" containerID="c79d56ad96b8f11afa49ae74505debb0d1df4399bd685e32614fb329fb662b3b" Jul 23 18:00:20.939231 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.939185 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79d56ad96b8f11afa49ae74505debb0d1df4399bd685e32614fb329fb662b3b"} err="failed to get container status \"c79d56ad96b8f11afa49ae74505debb0d1df4399bd685e32614fb329fb662b3b\": rpc error: code = NotFound desc = could not find container \"c79d56ad96b8f11afa49ae74505debb0d1df4399bd685e32614fb329fb662b3b\": container with ID starting with c79d56ad96b8f11afa49ae74505debb0d1df4399bd685e32614fb329fb662b3b not found: ID does not exist" Jul 23 18:00:20.953222 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.953199 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c98884d87-zhs2f"] Jul 23 18:00:20.957369 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:20.957344 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c98884d87-zhs2f"] Jul 23 18:00:21.117637 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:00:21.117604 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84e84ea4-effc-4fa6-8fbe-cc37667bb3dc" path="/var/lib/kubelet/pods/84e84ea4-effc-4fa6-8fbe-cc37667bb3dc/volumes" Jul 23 18:01:10.297828 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.297775 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-575c5bbc59-bdrlt"] Jul 23 18:01:10.298294 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.298098 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ac336c7-36ca-40a4-942e-7f11cb4590bb" containerName="console" Jul 23 18:01:10.298294 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.298110 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ac336c7-36ca-40a4-942e-7f11cb4590bb" containerName="console" Jul 23 18:01:10.298294 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.298125 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e84ea4-effc-4fa6-8fbe-cc37667bb3dc" containerName="console" Jul 23 18:01:10.298294 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.298131 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e84ea4-effc-4fa6-8fbe-cc37667bb3dc" containerName="console" Jul 23 18:01:10.298294 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.298179 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84e84ea4-effc-4fa6-8fbe-cc37667bb3dc" containerName="console" Jul 23 18:01:10.298294 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.298189 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ac336c7-36ca-40a4-942e-7f11cb4590bb" containerName="console" Jul 23 18:01:10.299983 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.299967 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.314623 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.314594 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-575c5bbc59-bdrlt"] Jul 23 18:01:10.393836 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.393788 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-trusted-ca-bundle\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.393929 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.393873 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-oauth-config\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.393929 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.393906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-serving-cert\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.394005 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.393937 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-oauth-serving-cert\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.394005 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.393979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-service-ca\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.394005 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.393999 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnmw\" (UniqueName: \"kubernetes.io/projected/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-kube-api-access-xgnmw\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.394103 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.394031 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-config\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.495028 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.494991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnmw\" (UniqueName: \"kubernetes.io/projected/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-kube-api-access-xgnmw\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.495219 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.495038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-config\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.495219 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.495066 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-trusted-ca-bundle\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.495219 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.495107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-oauth-config\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.495219 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.495139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-serving-cert\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.495219 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.495176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-oauth-serving-cert\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.495219 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.495210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-service-ca\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.495982 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.495953 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-trusted-ca-bundle\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.496089 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.495955 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-service-ca\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.496089 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.495955 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-config\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.496089 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.496044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-oauth-serving-cert\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.497532 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.497501 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-oauth-config\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.497686 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.497666 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-serving-cert\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.513831 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.513778 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnmw\" (UniqueName: \"kubernetes.io/projected/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-kube-api-access-xgnmw\") pod \"console-575c5bbc59-bdrlt\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.608542 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.608457 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:10.751379 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:10.751354 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-575c5bbc59-bdrlt"] Jul 23 18:01:10.753621 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:01:10.753597 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f8ba0e_7b1c_478a_8ba5_f2c75c1b9ba8.slice/crio-69a1db27858760893603ba882d90fa2e3384605562e50cdb758d85b620c01668 WatchSource:0}: Error finding container 69a1db27858760893603ba882d90fa2e3384605562e50cdb758d85b620c01668: Status 404 returned error can't find the container with id 69a1db27858760893603ba882d90fa2e3384605562e50cdb758d85b620c01668 Jul 23 18:01:11.125131 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:11.125099 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575c5bbc59-bdrlt" event={"ID":"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8","Type":"ContainerStarted","Data":"c1134fe2edc86c30b836e36c664c7d06d87b2d1e7db5dd43d2cb110986197841"} Jul 23 18:01:11.125131 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:11.125145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575c5bbc59-bdrlt" event={"ID":"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8","Type":"ContainerStarted","Data":"69a1db27858760893603ba882d90fa2e3384605562e50cdb758d85b620c01668"} Jul 23 18:01:20.609590 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:20.609541 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:20.610014 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:20.609626 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:20.614282 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:20.614265 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:20.633403 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:20.633332 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-575c5bbc59-bdrlt" podStartSLOduration=10.633318761 podStartE2EDuration="10.633318761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:01:11.152250593 +0000 UTC m=+220.593749211" watchObservedRunningTime="2026-07-23 18:01:20.633318761 +0000 UTC m=+230.074817354" Jul 23 18:01:21.170863 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:21.170837 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:01:21.231334 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:21.231302 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d8d9f6f9d-w7qxx"] Jul 23 18:01:46.253383 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.253310 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d8d9f6f9d-w7qxx" podUID="88e2eae4-dbfb-4955-8f79-6efd44616f8a" containerName="console" containerID="cri-o://0aacf7272221deaaa52c79e107c73e9e0125b13e993879d1757b1979313b5d13" gracePeriod=15 Jul 23 18:01:46.490603 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.490577 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 18:01:46.603968 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.603877 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-config\") pod \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " Jul 23 18:01:46.603968 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.603927 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-oauth-serving-cert\") pod \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " Jul 23 18:01:46.603968 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.603960 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-serving-cert\") pod \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " Jul 23 18:01:46.604244 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.604068 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-service-ca\") pod \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " Jul 23 18:01:46.604244 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.604122 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-oauth-config\") pod \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " Jul 23 18:01:46.604244 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.604147 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62c6t\" (UniqueName: \"kubernetes.io/projected/88e2eae4-dbfb-4955-8f79-6efd44616f8a-kube-api-access-62c6t\") pod \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " Jul 23 18:01:46.604404 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.604338 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-config" (OuterVolumeSpecName: "console-config") pod "88e2eae4-dbfb-4955-8f79-6efd44616f8a" (UID: "88e2eae4-dbfb-4955-8f79-6efd44616f8a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:01:46.604404 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.604344 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "88e2eae4-dbfb-4955-8f79-6efd44616f8a" (UID: "88e2eae4-dbfb-4955-8f79-6efd44616f8a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:01:46.604404 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.604344 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-trusted-ca-bundle\") pod \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\" (UID: \"88e2eae4-dbfb-4955-8f79-6efd44616f8a\") " Jul 23 18:01:46.604404 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.604391 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-service-ca" (OuterVolumeSpecName: "service-ca") pod "88e2eae4-dbfb-4955-8f79-6efd44616f8a" (UID: "88e2eae4-dbfb-4955-8f79-6efd44616f8a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:01:46.604694 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.604671 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "88e2eae4-dbfb-4955-8f79-6efd44616f8a" (UID: "88e2eae4-dbfb-4955-8f79-6efd44616f8a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:01:46.604781 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.604701 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-config\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:01:46.604781 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.604722 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-oauth-serving-cert\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:01:46.604781 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.604735 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-service-ca\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:01:46.606134 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.606103 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "88e2eae4-dbfb-4955-8f79-6efd44616f8a" (UID: "88e2eae4-dbfb-4955-8f79-6efd44616f8a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:01:46.606572 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.606552 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "88e2eae4-dbfb-4955-8f79-6efd44616f8a" (UID: "88e2eae4-dbfb-4955-8f79-6efd44616f8a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:01:46.606640 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.606582 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e2eae4-dbfb-4955-8f79-6efd44616f8a-kube-api-access-62c6t" (OuterVolumeSpecName: "kube-api-access-62c6t") pod "88e2eae4-dbfb-4955-8f79-6efd44616f8a" (UID: "88e2eae4-dbfb-4955-8f79-6efd44616f8a"). InnerVolumeSpecName "kube-api-access-62c6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:01:46.705373 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.705334 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-oauth-config\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:01:46.705373 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.705363 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62c6t\" (UniqueName: \"kubernetes.io/projected/88e2eae4-dbfb-4955-8f79-6efd44616f8a-kube-api-access-62c6t\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:01:46.705373 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.705379 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e2eae4-dbfb-4955-8f79-6efd44616f8a-trusted-ca-bundle\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:01:46.705607 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:46.705392 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88e2eae4-dbfb-4955-8f79-6efd44616f8a-console-serving-cert\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:01:47.269904 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:47.269867 2564 generic.go:356] "Generic (PLEG): container finished" podID="88e2eae4-dbfb-4955-8f79-6efd44616f8a" containerID="0aacf7272221deaaa52c79e107c73e9e0125b13e993879d1757b1979313b5d13" exitCode=0 Jul 23 18:01:47.270317 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:47.269921 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d8d9f6f9d-w7qxx" Jul 23 18:01:47.270317 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:47.269937 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d8d9f6f9d-w7qxx" event={"ID":"88e2eae4-dbfb-4955-8f79-6efd44616f8a","Type":"ContainerDied","Data":"0aacf7272221deaaa52c79e107c73e9e0125b13e993879d1757b1979313b5d13"} Jul 23 18:01:47.270317 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:47.269987 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d8d9f6f9d-w7qxx" event={"ID":"88e2eae4-dbfb-4955-8f79-6efd44616f8a","Type":"ContainerDied","Data":"7bea4186a367a8ef387afcd06ac23402028fd0981092648a2a9d810feb1c3a59"} Jul 23 18:01:47.270317 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:47.270017 2564 scope.go:117] "RemoveContainer" containerID="0aacf7272221deaaa52c79e107c73e9e0125b13e993879d1757b1979313b5d13" Jul 23 18:01:47.278104 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:47.278086 2564 scope.go:117] "RemoveContainer" containerID="0aacf7272221deaaa52c79e107c73e9e0125b13e993879d1757b1979313b5d13" Jul 23 18:01:47.278348 ip-10-0-142-147 kubenswrapper[2564]: E0723 18:01:47.278328 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aacf7272221deaaa52c79e107c73e9e0125b13e993879d1757b1979313b5d13\": container with ID starting with 0aacf7272221deaaa52c79e107c73e9e0125b13e993879d1757b1979313b5d13 not found: ID does not exist" containerID="0aacf7272221deaaa52c79e107c73e9e0125b13e993879d1757b1979313b5d13" Jul 23 18:01:47.278407 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:47.278360 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aacf7272221deaaa52c79e107c73e9e0125b13e993879d1757b1979313b5d13"} err="failed to get container status \"0aacf7272221deaaa52c79e107c73e9e0125b13e993879d1757b1979313b5d13\": rpc error: code = NotFound desc = could not find container \"0aacf7272221deaaa52c79e107c73e9e0125b13e993879d1757b1979313b5d13\": container with ID starting with 0aacf7272221deaaa52c79e107c73e9e0125b13e993879d1757b1979313b5d13 not found: ID does not exist" Jul 23 18:01:47.293340 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:47.293316 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d8d9f6f9d-w7qxx"] Jul 23 18:01:47.297038 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:47.297019 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d8d9f6f9d-w7qxx"] Jul 23 18:01:49.116916 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:01:49.116882 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e2eae4-dbfb-4955-8f79-6efd44616f8a" path="/var/lib/kubelet/pods/88e2eae4-dbfb-4955-8f79-6efd44616f8a/volumes" Jul 23 18:02:01.806870 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.806760 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bbbs4"] Jul 23 18:02:01.807314 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.807267 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e2eae4-dbfb-4955-8f79-6efd44616f8a" containerName="console" Jul 23 18:02:01.807314 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.807285 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e2eae4-dbfb-4955-8f79-6efd44616f8a" containerName="console" Jul 23 18:02:01.807416 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.807369 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e2eae4-dbfb-4955-8f79-6efd44616f8a" containerName="console" Jul 23 18:02:01.810344 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.810323 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bbbs4" Jul 23 18:02:01.818257 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.818238 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 23 18:02:01.818357 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.818244 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 23 18:02:01.818407 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.818371 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-cxk9s\"" Jul 23 18:02:01.821798 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.821774 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zfrs\" (UniqueName: \"kubernetes.io/projected/555fa58a-5d19-4bef-833d-89863045f846-kube-api-access-8zfrs\") pod \"cert-manager-operator-controller-manager-55c444bf86-bbbs4\" (UID: \"555fa58a-5d19-4bef-833d-89863045f846\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bbbs4" Jul 23 18:02:01.821914 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.821870 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/555fa58a-5d19-4bef-833d-89863045f846-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bbbs4\" (UID: \"555fa58a-5d19-4bef-833d-89863045f846\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bbbs4" Jul 23 18:02:01.841824 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.841785 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bbbs4"] Jul 23 18:02:01.923189 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.923150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfrs\" (UniqueName: \"kubernetes.io/projected/555fa58a-5d19-4bef-833d-89863045f846-kube-api-access-8zfrs\") pod \"cert-manager-operator-controller-manager-55c444bf86-bbbs4\" (UID: \"555fa58a-5d19-4bef-833d-89863045f846\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bbbs4" Jul 23 18:02:01.923349 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.923206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/555fa58a-5d19-4bef-833d-89863045f846-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bbbs4\" (UID: \"555fa58a-5d19-4bef-833d-89863045f846\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bbbs4" Jul 23 18:02:01.923578 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.923554 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/555fa58a-5d19-4bef-833d-89863045f846-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bbbs4\" (UID: \"555fa58a-5d19-4bef-833d-89863045f846\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bbbs4" Jul 23 18:02:01.942544 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:01.942508 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zfrs\" (UniqueName: \"kubernetes.io/projected/555fa58a-5d19-4bef-833d-89863045f846-kube-api-access-8zfrs\") pod \"cert-manager-operator-controller-manager-55c444bf86-bbbs4\" (UID: \"555fa58a-5d19-4bef-833d-89863045f846\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bbbs4" Jul 23 18:02:02.119347 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:02.119264 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bbbs4" Jul 23 18:02:02.276482 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:02.276452 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bbbs4"] Jul 23 18:02:02.280289 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:02:02.280263 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555fa58a_5d19_4bef_833d_89863045f846.slice/crio-2a8c439fa84480e0a2bad917bf016cc0be4269fc884bb9aa13cafa83fdf2896d WatchSource:0}: Error finding container 2a8c439fa84480e0a2bad917bf016cc0be4269fc884bb9aa13cafa83fdf2896d: Status 404 returned error can't find the container with id 2a8c439fa84480e0a2bad917bf016cc0be4269fc884bb9aa13cafa83fdf2896d Jul 23 18:02:02.329585 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:02.329551 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bbbs4" event={"ID":"555fa58a-5d19-4bef-833d-89863045f846","Type":"ContainerStarted","Data":"2a8c439fa84480e0a2bad917bf016cc0be4269fc884bb9aa13cafa83fdf2896d"} Jul 23 18:02:06.348200 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:06.348164 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bbbs4" event={"ID":"555fa58a-5d19-4bef-833d-89863045f846","Type":"ContainerStarted","Data":"a59121c734a0079cdc112835e52bd6c0005de596c003c191a061163dc4ef3610"} Jul 23 18:02:06.375482 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:06.375431 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bbbs4" podStartSLOduration=2.150826504 podStartE2EDuration="5.375414515s" totalImagesPullingTime="3.224588011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:02:06.372893053 +0000 UTC m=+275.814391646" watchObservedRunningTime="2026-07-23 18:02:06.375414515 +0000 UTC m=+275.816913108" Jul 23 18:02:07.285338 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.285297 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv"] Jul 23 18:02:07.288981 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.288956 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" Jul 23 18:02:07.295797 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.295776 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 18:02:07.296136 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.296118 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8np6d\"" Jul 23 18:02:07.296348 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.296330 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 18:02:07.315713 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.315684 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv"] Jul 23 18:02:07.371878 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.371845 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e27cea3a-16e0-4021-8abc-9e9e9b09b410-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv\" (UID: \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" Jul 23 18:02:07.372259 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.371910 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrtlj\" (UniqueName: \"kubernetes.io/projected/e27cea3a-16e0-4021-8abc-9e9e9b09b410-kube-api-access-vrtlj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv\" (UID: \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" Jul 23 18:02:07.372259 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.371935 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e27cea3a-16e0-4021-8abc-9e9e9b09b410-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv\" (UID: \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" Jul 23 18:02:07.473332 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.473297 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e27cea3a-16e0-4021-8abc-9e9e9b09b410-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv\" (UID: \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" Jul 23 18:02:07.473493 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.473430 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrtlj\" (UniqueName: \"kubernetes.io/projected/e27cea3a-16e0-4021-8abc-9e9e9b09b410-kube-api-access-vrtlj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv\" (UID: \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" Jul 23 18:02:07.473493 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.473470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e27cea3a-16e0-4021-8abc-9e9e9b09b410-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv\" (UID: \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" Jul 23 18:02:07.473758 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.473734 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e27cea3a-16e0-4021-8abc-9e9e9b09b410-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv\" (UID: \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" Jul 23 18:02:07.473859 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.473841 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e27cea3a-16e0-4021-8abc-9e9e9b09b410-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv\" (UID: \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" Jul 23 18:02:07.493052 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.493022 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrtlj\" (UniqueName: \"kubernetes.io/projected/e27cea3a-16e0-4021-8abc-9e9e9b09b410-kube-api-access-vrtlj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv\" (UID: \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" Jul 23 18:02:07.598521 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.598437 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" Jul 23 18:02:07.737939 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:07.737909 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv"] Jul 23 18:02:07.738171 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:02:07.738150 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27cea3a_16e0_4021_8abc_9e9e9b09b410.slice/crio-402c2cd7686a07679c731a7c81beaca954ac6bd22877a371ec48aa9c6a61fa8f WatchSource:0}: Error finding container 402c2cd7686a07679c731a7c81beaca954ac6bd22877a371ec48aa9c6a61fa8f: Status 404 returned error can't find the container with id 402c2cd7686a07679c731a7c81beaca954ac6bd22877a371ec48aa9c6a61fa8f Jul 23 18:02:08.359742 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:08.359709 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" event={"ID":"e27cea3a-16e0-4021-8abc-9e9e9b09b410","Type":"ContainerStarted","Data":"402c2cd7686a07679c731a7c81beaca954ac6bd22877a371ec48aa9c6a61fa8f"} Jul 23 18:02:14.388913 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:14.388870 2564 generic.go:356] "Generic (PLEG): container finished" podID="e27cea3a-16e0-4021-8abc-9e9e9b09b410" containerID="4619c5ea8ad276fcbc5e32c48ea7ce23e75e40b4248c3744993af0bab0b193da" exitCode=0 Jul 23 18:02:14.389297 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:14.388918 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" event={"ID":"e27cea3a-16e0-4021-8abc-9e9e9b09b410","Type":"ContainerDied","Data":"4619c5ea8ad276fcbc5e32c48ea7ce23e75e40b4248c3744993af0bab0b193da"} Jul 23 18:02:17.402749 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:17.402711 2564 generic.go:356] "Generic (PLEG): container finished" podID="e27cea3a-16e0-4021-8abc-9e9e9b09b410" containerID="8db2fb07d2160734501850ce94925e82d1b8f6a5c492c19b5aadf2bcaad2c57e" exitCode=0 Jul 23 18:02:17.403230 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:17.402796 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" event={"ID":"e27cea3a-16e0-4021-8abc-9e9e9b09b410","Type":"ContainerDied","Data":"8db2fb07d2160734501850ce94925e82d1b8f6a5c492c19b5aadf2bcaad2c57e"} Jul 23 18:02:25.442642 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:25.442557 2564 generic.go:356] "Generic (PLEG): container finished" podID="e27cea3a-16e0-4021-8abc-9e9e9b09b410" containerID="d18b99eef461e05d247eab54024cd4f5dccdfcd548512dd98c9db0baddfe8ace" exitCode=0 Jul 23 18:02:25.443027 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:25.442648 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" event={"ID":"e27cea3a-16e0-4021-8abc-9e9e9b09b410","Type":"ContainerDied","Data":"d18b99eef461e05d247eab54024cd4f5dccdfcd548512dd98c9db0baddfe8ace"} Jul 23 18:02:26.577145 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:26.577120 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" Jul 23 18:02:26.651118 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:26.651083 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e27cea3a-16e0-4021-8abc-9e9e9b09b410-util\") pod \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\" (UID: \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\") " Jul 23 18:02:26.651302 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:26.651134 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e27cea3a-16e0-4021-8abc-9e9e9b09b410-bundle\") pod \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\" (UID: \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\") " Jul 23 18:02:26.651302 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:26.651161 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrtlj\" (UniqueName: \"kubernetes.io/projected/e27cea3a-16e0-4021-8abc-9e9e9b09b410-kube-api-access-vrtlj\") pod \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\" (UID: \"e27cea3a-16e0-4021-8abc-9e9e9b09b410\") " Jul 23 18:02:26.651577 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:26.651550 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27cea3a-16e0-4021-8abc-9e9e9b09b410-bundle" (OuterVolumeSpecName: "bundle") pod "e27cea3a-16e0-4021-8abc-9e9e9b09b410" (UID: "e27cea3a-16e0-4021-8abc-9e9e9b09b410"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:02:26.653378 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:26.653351 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27cea3a-16e0-4021-8abc-9e9e9b09b410-kube-api-access-vrtlj" (OuterVolumeSpecName: "kube-api-access-vrtlj") pod "e27cea3a-16e0-4021-8abc-9e9e9b09b410" (UID: "e27cea3a-16e0-4021-8abc-9e9e9b09b410"). InnerVolumeSpecName "kube-api-access-vrtlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:02:26.655298 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:26.655273 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27cea3a-16e0-4021-8abc-9e9e9b09b410-util" (OuterVolumeSpecName: "util") pod "e27cea3a-16e0-4021-8abc-9e9e9b09b410" (UID: "e27cea3a-16e0-4021-8abc-9e9e9b09b410"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:02:26.752289 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:26.752260 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e27cea3a-16e0-4021-8abc-9e9e9b09b410-util\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:02:26.752289 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:26.752284 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e27cea3a-16e0-4021-8abc-9e9e9b09b410-bundle\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:02:26.752289 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:26.752293 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrtlj\" (UniqueName: \"kubernetes.io/projected/e27cea3a-16e0-4021-8abc-9e9e9b09b410-kube-api-access-vrtlj\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:02:27.454838 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:27.454740 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" event={"ID":"e27cea3a-16e0-4021-8abc-9e9e9b09b410","Type":"ContainerDied","Data":"402c2cd7686a07679c731a7c81beaca954ac6bd22877a371ec48aa9c6a61fa8f"} Jul 23 18:02:27.454838 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:27.454781 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftrdsv" Jul 23 18:02:27.454838 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:27.454787 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="402c2cd7686a07679c731a7c81beaca954ac6bd22877a371ec48aa9c6a61fa8f" Jul 23 18:02:30.996267 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:30.996238 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 18:02:51.057734 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.057698 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5"] Jul 23 18:02:51.059312 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.058036 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27cea3a-16e0-4021-8abc-9e9e9b09b410" containerName="util" Jul 23 18:02:51.059312 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.058047 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27cea3a-16e0-4021-8abc-9e9e9b09b410" containerName="util" Jul 23 18:02:51.059312 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.058055 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27cea3a-16e0-4021-8abc-9e9e9b09b410" containerName="extract" Jul 23 18:02:51.059312 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.058060 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27cea3a-16e0-4021-8abc-9e9e9b09b410" containerName="extract" Jul 23 18:02:51.059312 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.058076 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27cea3a-16e0-4021-8abc-9e9e9b09b410" containerName="pull" Jul 23 18:02:51.059312 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.058082 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27cea3a-16e0-4021-8abc-9e9e9b09b410" containerName="pull" Jul 23 18:02:51.059312 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.058168 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e27cea3a-16e0-4021-8abc-9e9e9b09b410" containerName="extract" Jul 23 18:02:51.060062 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.060046 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.063289 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.063261 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 18:02:51.063414 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.063269 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 18:02:51.063530 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.063513 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-z5mhz\"" Jul 23 18:02:51.063927 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.063908 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 18:02:51.064016 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.063995 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 18:02:51.152440 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.152407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79ee29b5-8549-413b-af02-2621002d17e2-tmp\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.152633 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.152464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/79ee29b5-8549-413b-af02-2621002d17e2-manifests\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.152633 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.152523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d44nq\" (UniqueName: \"kubernetes.io/projected/79ee29b5-8549-413b-af02-2621002d17e2-kube-api-access-d44nq\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.152633 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.152568 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/79ee29b5-8549-413b-af02-2621002d17e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.152633 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.152602 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79ee29b5-8549-413b-af02-2621002d17e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.167881 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.167852 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5"] Jul 23 18:02:51.253227 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.253184 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/79ee29b5-8549-413b-af02-2621002d17e2-manifests\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.253420 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.253234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d44nq\" (UniqueName: \"kubernetes.io/projected/79ee29b5-8549-413b-af02-2621002d17e2-kube-api-access-d44nq\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.253420 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.253266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/79ee29b5-8549-413b-af02-2621002d17e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.253420 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.253301 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79ee29b5-8549-413b-af02-2621002d17e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.253420 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.253350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79ee29b5-8549-413b-af02-2621002d17e2-tmp\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.253663 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.253639 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/79ee29b5-8549-413b-af02-2621002d17e2-manifests\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.255646 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.255614 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79ee29b5-8549-413b-af02-2621002d17e2-tmp\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.255897 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.255729 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/79ee29b5-8549-413b-af02-2621002d17e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.255995 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.255962 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79ee29b5-8549-413b-af02-2621002d17e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.272087 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.272062 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d44nq\" (UniqueName: \"kubernetes.io/projected/79ee29b5-8549-413b-af02-2621002d17e2-kube-api-access-d44nq\") pod \"opendatahub-operator-controller-manager-9f47b6774-kb9x5\" (UID: \"79ee29b5-8549-413b-af02-2621002d17e2\") " pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.370334 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.370246 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:51.509099 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.509071 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5"] Jul 23 18:02:51.510782 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:02:51.510751 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ee29b5_8549_413b_af02_2621002d17e2.slice/crio-1ac4b0170a407a13651801be8bb245bfa15283f579d100ba27c75f647faf2e46 WatchSource:0}: Error finding container 1ac4b0170a407a13651801be8bb245bfa15283f579d100ba27c75f647faf2e46: Status 404 returned error can't find the container with id 1ac4b0170a407a13651801be8bb245bfa15283f579d100ba27c75f647faf2e46 Jul 23 18:02:51.512586 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.512570 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:02:51.555077 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:51.555045 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" event={"ID":"79ee29b5-8549-413b-af02-2621002d17e2","Type":"ContainerStarted","Data":"1ac4b0170a407a13651801be8bb245bfa15283f579d100ba27c75f647faf2e46"} Jul 23 18:02:55.577320 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:55.577283 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" event={"ID":"79ee29b5-8549-413b-af02-2621002d17e2","Type":"ContainerStarted","Data":"52a9fc8d68a902ae781f8a500f3eadc2fec2abb54fd84eb3f0fdd82187989666"} Jul 23 18:02:57.588241 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:57.588204 2564 generic.go:356] "Generic (PLEG): container finished" podID="79ee29b5-8549-413b-af02-2621002d17e2" containerID="52a9fc8d68a902ae781f8a500f3eadc2fec2abb54fd84eb3f0fdd82187989666" exitCode=0 Jul 23 18:02:57.588673 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:57.588267 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" event={"ID":"79ee29b5-8549-413b-af02-2621002d17e2","Type":"ContainerDied","Data":"52a9fc8d68a902ae781f8a500f3eadc2fec2abb54fd84eb3f0fdd82187989666"} Jul 23 18:02:58.594297 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:58.594258 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" event={"ID":"79ee29b5-8549-413b-af02-2621002d17e2","Type":"ContainerStarted","Data":"1ee30664626bbfacef2638239d487ff305c3a004c2707436867ea447f688e091"} Jul 23 18:02:58.594837 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:58.594494 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:02:58.730340 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:02:58.730280 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" podStartSLOduration=2.515106806 podStartE2EDuration="7.730259248s" totalImagesPullingTime="4.181872878s" totalInitContainerRuntime="1.033279564s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:02:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:02:58.726240117 +0000 UTC m=+328.167738729" watchObservedRunningTime="2026-07-23 18:02:58.730259248 +0000 UTC m=+328.171757843" Jul 23 18:03:09.601287 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:09.601255 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9f47b6774-kb9x5" Jul 23 18:03:11.044594 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.044553 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd"] Jul 23 18:03:11.050349 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.050245 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:11.054661 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.054640 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 18:03:11.056040 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.056020 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 18:03:11.056204 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.056039 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 18:03:11.056204 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.056062 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 18:03:11.056204 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.056067 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 18:03:11.056204 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.056027 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-b6cs4\"" Jul 23 18:03:11.072572 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.072530 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd"] Jul 23 18:03:11.127445 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.127409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/45bc571d-84a5-4af4-aa85-49b1b313371c-metrics-cert\") pod \"lws-controller-manager-56f479bbd7-2mdsd\" (UID: \"45bc571d-84a5-4af4-aa85-49b1b313371c\") " pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:11.127628 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.127456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45bc571d-84a5-4af4-aa85-49b1b313371c-cert\") pod \"lws-controller-manager-56f479bbd7-2mdsd\" (UID: \"45bc571d-84a5-4af4-aa85-49b1b313371c\") " pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:11.127628 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.127512 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/45bc571d-84a5-4af4-aa85-49b1b313371c-manager-config\") pod \"lws-controller-manager-56f479bbd7-2mdsd\" (UID: \"45bc571d-84a5-4af4-aa85-49b1b313371c\") " pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:11.127628 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.127529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g79wl\" (UniqueName: \"kubernetes.io/projected/45bc571d-84a5-4af4-aa85-49b1b313371c-kube-api-access-g79wl\") pod \"lws-controller-manager-56f479bbd7-2mdsd\" (UID: \"45bc571d-84a5-4af4-aa85-49b1b313371c\") " pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:11.228405 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.228372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45bc571d-84a5-4af4-aa85-49b1b313371c-cert\") pod \"lws-controller-manager-56f479bbd7-2mdsd\" (UID: \"45bc571d-84a5-4af4-aa85-49b1b313371c\") " pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:11.228578 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.228423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/45bc571d-84a5-4af4-aa85-49b1b313371c-manager-config\") pod \"lws-controller-manager-56f479bbd7-2mdsd\" (UID: \"45bc571d-84a5-4af4-aa85-49b1b313371c\") " pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:11.228578 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.228449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g79wl\" (UniqueName: \"kubernetes.io/projected/45bc571d-84a5-4af4-aa85-49b1b313371c-kube-api-access-g79wl\") pod \"lws-controller-manager-56f479bbd7-2mdsd\" (UID: \"45bc571d-84a5-4af4-aa85-49b1b313371c\") " pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:11.228578 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.228521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/45bc571d-84a5-4af4-aa85-49b1b313371c-metrics-cert\") pod \"lws-controller-manager-56f479bbd7-2mdsd\" (UID: \"45bc571d-84a5-4af4-aa85-49b1b313371c\") " pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:11.229161 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.229133 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/45bc571d-84a5-4af4-aa85-49b1b313371c-manager-config\") pod \"lws-controller-manager-56f479bbd7-2mdsd\" (UID: \"45bc571d-84a5-4af4-aa85-49b1b313371c\") " pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:11.230997 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.230974 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/45bc571d-84a5-4af4-aa85-49b1b313371c-metrics-cert\") pod \"lws-controller-manager-56f479bbd7-2mdsd\" (UID: \"45bc571d-84a5-4af4-aa85-49b1b313371c\") " pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:11.231076 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.231011 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45bc571d-84a5-4af4-aa85-49b1b313371c-cert\") pod \"lws-controller-manager-56f479bbd7-2mdsd\" (UID: \"45bc571d-84a5-4af4-aa85-49b1b313371c\") " pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:11.240708 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.240680 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g79wl\" (UniqueName: \"kubernetes.io/projected/45bc571d-84a5-4af4-aa85-49b1b313371c-kube-api-access-g79wl\") pod \"lws-controller-manager-56f479bbd7-2mdsd\" (UID: \"45bc571d-84a5-4af4-aa85-49b1b313371c\") " pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:11.359775 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.359689 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:11.520325 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.520296 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd"] Jul 23 18:03:11.522902 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:03:11.522875 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bc571d_84a5_4af4_aa85_49b1b313371c.slice/crio-9d4f0e46d14ebb27bde0cdd0207dd0e35b5d3e78a7a8150613b41629c94fd4b3 WatchSource:0}: Error finding container 9d4f0e46d14ebb27bde0cdd0207dd0e35b5d3e78a7a8150613b41629c94fd4b3: Status 404 returned error can't find the container with id 9d4f0e46d14ebb27bde0cdd0207dd0e35b5d3e78a7a8150613b41629c94fd4b3 Jul 23 18:03:11.655354 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:11.655270 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" event={"ID":"45bc571d-84a5-4af4-aa85-49b1b313371c","Type":"ContainerStarted","Data":"9d4f0e46d14ebb27bde0cdd0207dd0e35b5d3e78a7a8150613b41629c94fd4b3"} Jul 23 18:03:12.675914 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.675883 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm"] Jul 23 18:03:12.678518 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.678497 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm" Jul 23 18:03:12.683216 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.683156 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 18:03:12.684410 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.684391 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8d7c4\"" Jul 23 18:03:12.696227 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.696207 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 23 18:03:12.702548 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.702528 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm"] Jul 23 18:03:12.740261 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.740226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/961f48be-e367-4efd-bbeb-570bf06a5d68-tls-certs\") pod \"kube-auth-proxy-cdff8959d-5s8vm\" (UID: \"961f48be-e367-4efd-bbeb-570bf06a5d68\") " pod="openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm" Jul 23 18:03:12.740438 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.740329 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl2fp\" (UniqueName: \"kubernetes.io/projected/961f48be-e367-4efd-bbeb-570bf06a5d68-kube-api-access-zl2fp\") pod \"kube-auth-proxy-cdff8959d-5s8vm\" (UID: \"961f48be-e367-4efd-bbeb-570bf06a5d68\") " pod="openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm" Jul 23 18:03:12.740438 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.740396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/961f48be-e367-4efd-bbeb-570bf06a5d68-tmp\") pod \"kube-auth-proxy-cdff8959d-5s8vm\" (UID: \"961f48be-e367-4efd-bbeb-570bf06a5d68\") " pod="openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm" Jul 23 18:03:12.841593 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.841557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl2fp\" (UniqueName: \"kubernetes.io/projected/961f48be-e367-4efd-bbeb-570bf06a5d68-kube-api-access-zl2fp\") pod \"kube-auth-proxy-cdff8959d-5s8vm\" (UID: \"961f48be-e367-4efd-bbeb-570bf06a5d68\") " pod="openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm" Jul 23 18:03:12.841792 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.841634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/961f48be-e367-4efd-bbeb-570bf06a5d68-tmp\") pod \"kube-auth-proxy-cdff8959d-5s8vm\" (UID: \"961f48be-e367-4efd-bbeb-570bf06a5d68\") " pod="openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm" Jul 23 18:03:12.841792 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.841680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/961f48be-e367-4efd-bbeb-570bf06a5d68-tls-certs\") pod \"kube-auth-proxy-cdff8959d-5s8vm\" (UID: \"961f48be-e367-4efd-bbeb-570bf06a5d68\") " pod="openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm" Jul 23 18:03:12.844664 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.844591 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/961f48be-e367-4efd-bbeb-570bf06a5d68-tmp\") pod \"kube-auth-proxy-cdff8959d-5s8vm\" (UID: \"961f48be-e367-4efd-bbeb-570bf06a5d68\") " pod="openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm" Jul 23 18:03:12.844790 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.844757 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/961f48be-e367-4efd-bbeb-570bf06a5d68-tls-certs\") pod \"kube-auth-proxy-cdff8959d-5s8vm\" (UID: \"961f48be-e367-4efd-bbeb-570bf06a5d68\") " pod="openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm" Jul 23 18:03:12.879463 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.879419 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl2fp\" (UniqueName: \"kubernetes.io/projected/961f48be-e367-4efd-bbeb-570bf06a5d68-kube-api-access-zl2fp\") pod \"kube-auth-proxy-cdff8959d-5s8vm\" (UID: \"961f48be-e367-4efd-bbeb-570bf06a5d68\") " pod="openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm" Jul 23 18:03:12.990040 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:12.990007 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm" Jul 23 18:03:13.519793 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:03:13.519763 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961f48be_e367_4efd_bbeb_570bf06a5d68.slice/crio-8d9b3a198d62f82c8ee4884d0990135da88a78d05d6c225eeac0af474123d57f WatchSource:0}: Error finding container 8d9b3a198d62f82c8ee4884d0990135da88a78d05d6c225eeac0af474123d57f: Status 404 returned error can't find the container with id 8d9b3a198d62f82c8ee4884d0990135da88a78d05d6c225eeac0af474123d57f Jul 23 18:03:13.521462 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:13.521350 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm"] Jul 23 18:03:13.666402 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:13.666306 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm" event={"ID":"961f48be-e367-4efd-bbeb-570bf06a5d68","Type":"ContainerStarted","Data":"8d9b3a198d62f82c8ee4884d0990135da88a78d05d6c225eeac0af474123d57f"} Jul 23 18:03:13.667727 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:13.667702 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" event={"ID":"45bc571d-84a5-4af4-aa85-49b1b313371c","Type":"ContainerStarted","Data":"c1d78cd892fc57d2faf2481287c816ecac296e4aefdd125abf3c01a2bbd4b66b"} Jul 23 18:03:13.667853 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:13.667828 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:03:13.697964 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:13.697913 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" podStartSLOduration=0.864058031 podStartE2EDuration="2.697896636s" totalImagesPullingTime="1.833838605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:03:13.696639123 +0000 UTC m=+343.138137740" watchObservedRunningTime="2026-07-23 18:03:13.697896636 +0000 UTC m=+343.139395229" Jul 23 18:03:16.683126 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:16.683088 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm" event={"ID":"961f48be-e367-4efd-bbeb-570bf06a5d68","Type":"ContainerStarted","Data":"d27e949c0d786920bc970ea2f9f2cb9ddc40908b3f9e5cb479ae7f448f7e8510"} Jul 23 18:03:16.733119 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:16.733065 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cdff8959d-5s8vm" podStartSLOduration=1.711324464 podStartE2EDuration="4.73304911s" totalImagesPullingTime="3.021724646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:03:16.730620511 +0000 UTC m=+346.172119104" watchObservedRunningTime="2026-07-23 18:03:16.73304911 +0000 UTC m=+346.174547704" Jul 23 18:03:24.674039 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:03:24.673960 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-56f479bbd7-2mdsd" Jul 23 18:04:58.230888 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:04:58.230798 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-575c5bbc59-bdrlt"] Jul 23 18:05:00.161822 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.161775 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27"] Jul 23 18:05:00.165097 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.165078 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27" Jul 23 18:05:00.168181 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.168161 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 18:05:00.168318 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.168159 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 23 18:05:00.168318 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.168223 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 18:05:00.169433 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.169417 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2dhdm\"" Jul 23 18:05:00.169504 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.169421 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 23 18:05:00.189230 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.189201 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27"] Jul 23 18:05:00.269250 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.269213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79jhz\" (UniqueName: \"kubernetes.io/projected/2f969d3d-c3ed-45b8-b651-cfd4a64bb13a-kube-api-access-79jhz\") pod \"kuadrant-console-plugin-66b5774985-f4j27\" (UID: \"2f969d3d-c3ed-45b8-b651-cfd4a64bb13a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27" Jul 23 18:05:00.269413 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.269285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f969d3d-c3ed-45b8-b651-cfd4a64bb13a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-f4j27\" (UID: \"2f969d3d-c3ed-45b8-b651-cfd4a64bb13a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27" Jul 23 18:05:00.269413 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.269353 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2f969d3d-c3ed-45b8-b651-cfd4a64bb13a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-f4j27\" (UID: \"2f969d3d-c3ed-45b8-b651-cfd4a64bb13a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27" Jul 23 18:05:00.370351 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.370314 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f969d3d-c3ed-45b8-b651-cfd4a64bb13a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-f4j27\" (UID: \"2f969d3d-c3ed-45b8-b651-cfd4a64bb13a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27" Jul 23 18:05:00.370504 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.370377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2f969d3d-c3ed-45b8-b651-cfd4a64bb13a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-f4j27\" (UID: \"2f969d3d-c3ed-45b8-b651-cfd4a64bb13a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27" Jul 23 18:05:00.370504 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.370429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79jhz\" (UniqueName: \"kubernetes.io/projected/2f969d3d-c3ed-45b8-b651-cfd4a64bb13a-kube-api-access-79jhz\") pod \"kuadrant-console-plugin-66b5774985-f4j27\" (UID: \"2f969d3d-c3ed-45b8-b651-cfd4a64bb13a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27" Jul 23 18:05:00.371064 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.371037 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2f969d3d-c3ed-45b8-b651-cfd4a64bb13a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-f4j27\" (UID: \"2f969d3d-c3ed-45b8-b651-cfd4a64bb13a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27" Jul 23 18:05:00.372733 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.372708 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f969d3d-c3ed-45b8-b651-cfd4a64bb13a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-f4j27\" (UID: \"2f969d3d-c3ed-45b8-b651-cfd4a64bb13a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27" Jul 23 18:05:00.388726 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.388700 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79jhz\" (UniqueName: \"kubernetes.io/projected/2f969d3d-c3ed-45b8-b651-cfd4a64bb13a-kube-api-access-79jhz\") pod \"kuadrant-console-plugin-66b5774985-f4j27\" (UID: \"2f969d3d-c3ed-45b8-b651-cfd4a64bb13a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27" Jul 23 18:05:00.482610 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.482577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27" Jul 23 18:05:00.639921 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:00.639893 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27"] Jul 23 18:05:00.641454 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:05:00.641422 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f969d3d_c3ed_45b8_b651_cfd4a64bb13a.slice/crio-87df1ee1c732adf1abc7f4b2f9cc3f5f9b97dacac7e5c5851a5330926c48c720 WatchSource:0}: Error finding container 87df1ee1c732adf1abc7f4b2f9cc3f5f9b97dacac7e5c5851a5330926c48c720: Status 404 returned error can't find the container with id 87df1ee1c732adf1abc7f4b2f9cc3f5f9b97dacac7e5c5851a5330926c48c720 Jul 23 18:05:01.156096 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:01.156058 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27" event={"ID":"2f969d3d-c3ed-45b8-b651-cfd4a64bb13a","Type":"ContainerStarted","Data":"87df1ee1c732adf1abc7f4b2f9cc3f5f9b97dacac7e5c5851a5330926c48c720"} Jul 23 18:05:23.250365 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.250302 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-575c5bbc59-bdrlt" podUID="27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8" containerName="console" containerID="cri-o://c1134fe2edc86c30b836e36c664c7d06d87b2d1e7db5dd43d2cb110986197841" gracePeriod=15 Jul 23 18:05:23.560227 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.560200 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:05:23.584620 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.584588 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-trusted-ca-bundle\") pod \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " Jul 23 18:05:23.584620 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.584623 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-serving-cert\") pod \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " Jul 23 18:05:23.584824 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.584669 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-config\") pod \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " Jul 23 18:05:23.584824 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.584696 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-service-ca\") pod \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " Jul 23 18:05:23.584824 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.584720 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-oauth-config\") pod \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " Jul 23 18:05:23.584824 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.584779 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-oauth-serving-cert\") pod \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " Jul 23 18:05:23.585036 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.584829 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgnmw\" (UniqueName: \"kubernetes.io/projected/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-kube-api-access-xgnmw\") pod \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\" (UID: \"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8\") " Jul 23 18:05:23.585093 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.585044 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8" (UID: "27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:05:23.585154 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.585090 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-config" (OuterVolumeSpecName: "console-config") pod "27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8" (UID: "27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:05:23.585154 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.585102 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-trusted-ca-bundle\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:05:23.585154 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.585117 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-service-ca" (OuterVolumeSpecName: "service-ca") pod "27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8" (UID: "27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:05:23.585427 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.585391 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8" (UID: "27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:05:23.587066 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.587035 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-kube-api-access-xgnmw" (OuterVolumeSpecName: "kube-api-access-xgnmw") pod "27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8" (UID: "27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8"). InnerVolumeSpecName "kube-api-access-xgnmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:05:23.587066 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.587039 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8" (UID: "27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:05:23.587196 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.587106 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8" (UID: "27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:05:23.686009 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.685974 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-serving-cert\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:05:23.686009 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.686003 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-config\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:05:23.686009 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.686014 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-service-ca\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:05:23.686227 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.686022 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-console-oauth-config\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:05:23.686227 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.686032 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-oauth-serving-cert\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:05:23.686227 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:23.686040 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgnmw\" (UniqueName: \"kubernetes.io/projected/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8-kube-api-access-xgnmw\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:05:24.280240 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:24.280196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27" event={"ID":"2f969d3d-c3ed-45b8-b651-cfd4a64bb13a","Type":"ContainerStarted","Data":"df046680b516acf40ff93af9a3f95ea3d7c0a742f7d41d8ca0f9c0bc08d0e13f"} Jul 23 18:05:24.281491 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:24.281469 2564 generic.go:356] "Generic (PLEG): container finished" podID="27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8" containerID="c1134fe2edc86c30b836e36c664c7d06d87b2d1e7db5dd43d2cb110986197841" exitCode=0 Jul 23 18:05:24.281592 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:24.281503 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575c5bbc59-bdrlt" Jul 23 18:05:24.281592 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:24.281516 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575c5bbc59-bdrlt" event={"ID":"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8","Type":"ContainerDied","Data":"c1134fe2edc86c30b836e36c664c7d06d87b2d1e7db5dd43d2cb110986197841"} Jul 23 18:05:24.281592 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:24.281547 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575c5bbc59-bdrlt" event={"ID":"27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8","Type":"ContainerDied","Data":"69a1db27858760893603ba882d90fa2e3384605562e50cdb758d85b620c01668"} Jul 23 18:05:24.281592 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:24.281565 2564 scope.go:117] "RemoveContainer" containerID="c1134fe2edc86c30b836e36c664c7d06d87b2d1e7db5dd43d2cb110986197841" Jul 23 18:05:24.290156 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:24.290140 2564 scope.go:117] "RemoveContainer" containerID="c1134fe2edc86c30b836e36c664c7d06d87b2d1e7db5dd43d2cb110986197841" Jul 23 18:05:24.290373 ip-10-0-142-147 kubenswrapper[2564]: E0723 18:05:24.290343 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1134fe2edc86c30b836e36c664c7d06d87b2d1e7db5dd43d2cb110986197841\": container with ID starting with c1134fe2edc86c30b836e36c664c7d06d87b2d1e7db5dd43d2cb110986197841 not found: ID does not exist" containerID="c1134fe2edc86c30b836e36c664c7d06d87b2d1e7db5dd43d2cb110986197841" Jul 23 18:05:24.290432 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:24.290379 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1134fe2edc86c30b836e36c664c7d06d87b2d1e7db5dd43d2cb110986197841"} err="failed to get container status \"c1134fe2edc86c30b836e36c664c7d06d87b2d1e7db5dd43d2cb110986197841\": rpc error: code = NotFound desc = could not find container \"c1134fe2edc86c30b836e36c664c7d06d87b2d1e7db5dd43d2cb110986197841\": container with ID starting with c1134fe2edc86c30b836e36c664c7d06d87b2d1e7db5dd43d2cb110986197841 not found: ID does not exist" Jul 23 18:05:24.304230 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:24.304192 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f4j27" podStartSLOduration=1.159548455 podStartE2EDuration="24.304177901s" totalImagesPullingTime="23.144629446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:05:24.303726374 +0000 UTC m=+473.745224968" watchObservedRunningTime="2026-07-23 18:05:24.304177901 +0000 UTC m=+473.745676495" Jul 23 18:05:24.322208 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:24.322178 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-575c5bbc59-bdrlt"] Jul 23 18:05:24.326837 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:24.326794 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-575c5bbc59-bdrlt"] Jul 23 18:05:25.121716 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:25.121679 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8" path="/var/lib/kubelet/pods/27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8/volumes" Jul 23 18:05:43.658542 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.658504 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:05:43.659081 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.659046 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8" containerName="console" Jul 23 18:05:43.659081 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.659066 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8" containerName="console" Jul 23 18:05:43.659197 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.659153 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f8ba0e-7b1c-478a-8ba5-f2c75c1b9ba8" containerName="console" Jul 23 18:05:43.665341 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.665320 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-m2vcp" Jul 23 18:05:43.670455 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.670434 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 23 18:05:43.692091 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.692063 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:05:43.744630 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.744600 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:05:43.752364 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.752344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3463c88d-e270-434f-9e29-c5b6fe4ac468-config-file\") pod \"limitador-limitador-54d6d988d5-m2vcp\" (UID: \"3463c88d-e270-434f-9e29-c5b6fe4ac468\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m2vcp" Jul 23 18:05:43.752481 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.752399 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqk8c\" (UniqueName: \"kubernetes.io/projected/3463c88d-e270-434f-9e29-c5b6fe4ac468-kube-api-access-gqk8c\") pod \"limitador-limitador-54d6d988d5-m2vcp\" (UID: \"3463c88d-e270-434f-9e29-c5b6fe4ac468\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m2vcp" Jul 23 18:05:43.853149 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.853116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3463c88d-e270-434f-9e29-c5b6fe4ac468-config-file\") pod \"limitador-limitador-54d6d988d5-m2vcp\" (UID: \"3463c88d-e270-434f-9e29-c5b6fe4ac468\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m2vcp" Jul 23 18:05:43.853311 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.853178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqk8c\" (UniqueName: \"kubernetes.io/projected/3463c88d-e270-434f-9e29-c5b6fe4ac468-kube-api-access-gqk8c\") pod \"limitador-limitador-54d6d988d5-m2vcp\" (UID: \"3463c88d-e270-434f-9e29-c5b6fe4ac468\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m2vcp" Jul 23 18:05:43.853755 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.853735 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3463c88d-e270-434f-9e29-c5b6fe4ac468-config-file\") pod \"limitador-limitador-54d6d988d5-m2vcp\" (UID: \"3463c88d-e270-434f-9e29-c5b6fe4ac468\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m2vcp" Jul 23 18:05:43.862689 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.862659 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqk8c\" (UniqueName: \"kubernetes.io/projected/3463c88d-e270-434f-9e29-c5b6fe4ac468-kube-api-access-gqk8c\") pod \"limitador-limitador-54d6d988d5-m2vcp\" (UID: \"3463c88d-e270-434f-9e29-c5b6fe4ac468\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m2vcp" Jul 23 18:05:43.974938 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:43.974912 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-m2vcp" Jul 23 18:05:44.126232 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:44.126175 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:05:44.128596 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:05:44.128563 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3463c88d_e270_434f_9e29_c5b6fe4ac468.slice/crio-244b21c81bdf4e7c94a70a921c9f1c418d13f17aac81e9ec56809b4d1eb674bf WatchSource:0}: Error finding container 244b21c81bdf4e7c94a70a921c9f1c418d13f17aac81e9ec56809b4d1eb674bf: Status 404 returned error can't find the container with id 244b21c81bdf4e7c94a70a921c9f1c418d13f17aac81e9ec56809b4d1eb674bf Jul 23 18:05:44.375507 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:44.375426 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-m2vcp" event={"ID":"3463c88d-e270-434f-9e29-c5b6fe4ac468","Type":"ContainerStarted","Data":"244b21c81bdf4e7c94a70a921c9f1c418d13f17aac81e9ec56809b4d1eb674bf"} Jul 23 18:05:44.440087 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:44.440056 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-nv4dm"] Jul 23 18:05:44.443588 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:44.443572 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nv4dm" Jul 23 18:05:44.447243 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:44.447224 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lvr5d\"" Jul 23 18:05:44.457172 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:44.457148 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nv4dm"] Jul 23 18:05:44.559865 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:44.559827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kwjh\" (UniqueName: \"kubernetes.io/projected/e030935f-a095-42e7-8caa-8de8922825cf-kube-api-access-7kwjh\") pod \"authorino-75d556bd77-nv4dm\" (UID: \"e030935f-a095-42e7-8caa-8de8922825cf\") " pod="kuadrant-system/authorino-75d556bd77-nv4dm" Jul 23 18:05:44.660967 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:44.660886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kwjh\" (UniqueName: \"kubernetes.io/projected/e030935f-a095-42e7-8caa-8de8922825cf-kube-api-access-7kwjh\") pod \"authorino-75d556bd77-nv4dm\" (UID: \"e030935f-a095-42e7-8caa-8de8922825cf\") " pod="kuadrant-system/authorino-75d556bd77-nv4dm" Jul 23 18:05:44.671916 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:44.671885 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kwjh\" (UniqueName: \"kubernetes.io/projected/e030935f-a095-42e7-8caa-8de8922825cf-kube-api-access-7kwjh\") pod \"authorino-75d556bd77-nv4dm\" (UID: \"e030935f-a095-42e7-8caa-8de8922825cf\") " pod="kuadrant-system/authorino-75d556bd77-nv4dm" Jul 23 18:05:44.753464 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:44.753424 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nv4dm" Jul 23 18:05:44.939562 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:44.939000 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nv4dm"] Jul 23 18:05:44.941995 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:05:44.941925 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode030935f_a095_42e7_8caa_8de8922825cf.slice/crio-8f7aecf61c4a4b939b54e12d9539510534497012dcd87e2ca2c52aa2e4bfd8db WatchSource:0}: Error finding container 8f7aecf61c4a4b939b54e12d9539510534497012dcd87e2ca2c52aa2e4bfd8db: Status 404 returned error can't find the container with id 8f7aecf61c4a4b939b54e12d9539510534497012dcd87e2ca2c52aa2e4bfd8db Jul 23 18:05:45.382728 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:45.382675 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nv4dm" event={"ID":"e030935f-a095-42e7-8caa-8de8922825cf","Type":"ContainerStarted","Data":"8f7aecf61c4a4b939b54e12d9539510534497012dcd87e2ca2c52aa2e4bfd8db"} Jul 23 18:05:47.397292 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:47.397260 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-m2vcp" event={"ID":"3463c88d-e270-434f-9e29-c5b6fe4ac468","Type":"ContainerStarted","Data":"3bf90a30c4284b9b0423832219001071aacc091420a9f1f5633e1a72afd9a3bd"} Jul 23 18:05:47.397705 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:47.397346 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-m2vcp" Jul 23 18:05:47.425487 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:47.425137 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-m2vcp" podStartSLOduration=1.6843459219999999 podStartE2EDuration="4.425114471s" totalImagesPullingTime="2.740768549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:05:47.424935471 +0000 UTC m=+496.866434066" watchObservedRunningTime="2026-07-23 18:05:47.425114471 +0000 UTC m=+496.866613066" Jul 23 18:05:49.409020 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:49.408977 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nv4dm" event={"ID":"e030935f-a095-42e7-8caa-8de8922825cf","Type":"ContainerStarted","Data":"1553439d9a0c2070fd30b6c6a4d2a3cbf0cfc3e935f16613db40ed36aca1781e"} Jul 23 18:05:49.428206 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:49.428158 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-nv4dm" podStartSLOduration=1.146557866 podStartE2EDuration="5.428143328s" totalImagesPullingTime="4.281585462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:05:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:05:49.426542469 +0000 UTC m=+498.868041060" watchObservedRunningTime="2026-07-23 18:05:49.428143328 +0000 UTC m=+498.869641922" Jul 23 18:05:58.403850 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:05:58.403798 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-m2vcp" Jul 23 18:06:20.328556 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:06:20.328522 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nv4dm"] Jul 23 18:06:20.329007 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:06:20.328724 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-nv4dm" podUID="e030935f-a095-42e7-8caa-8de8922825cf" containerName="authorino" containerID="cri-o://1553439d9a0c2070fd30b6c6a4d2a3cbf0cfc3e935f16613db40ed36aca1781e" gracePeriod=30 Jul 23 18:06:20.552174 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:06:20.552142 2564 generic.go:356] "Generic (PLEG): container finished" podID="e030935f-a095-42e7-8caa-8de8922825cf" containerID="1553439d9a0c2070fd30b6c6a4d2a3cbf0cfc3e935f16613db40ed36aca1781e" exitCode=0 Jul 23 18:06:20.552335 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:06:20.552199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nv4dm" event={"ID":"e030935f-a095-42e7-8caa-8de8922825cf","Type":"ContainerDied","Data":"1553439d9a0c2070fd30b6c6a4d2a3cbf0cfc3e935f16613db40ed36aca1781e"} Jul 23 18:06:20.580141 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:06:20.580071 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nv4dm" Jul 23 18:06:20.674191 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:06:20.674155 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kwjh\" (UniqueName: \"kubernetes.io/projected/e030935f-a095-42e7-8caa-8de8922825cf-kube-api-access-7kwjh\") pod \"e030935f-a095-42e7-8caa-8de8922825cf\" (UID: \"e030935f-a095-42e7-8caa-8de8922825cf\") " Jul 23 18:06:20.676372 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:06:20.676352 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e030935f-a095-42e7-8caa-8de8922825cf-kube-api-access-7kwjh" (OuterVolumeSpecName: "kube-api-access-7kwjh") pod "e030935f-a095-42e7-8caa-8de8922825cf" (UID: "e030935f-a095-42e7-8caa-8de8922825cf"). InnerVolumeSpecName "kube-api-access-7kwjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:06:20.775560 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:06:20.775521 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kwjh\" (UniqueName: \"kubernetes.io/projected/e030935f-a095-42e7-8caa-8de8922825cf-kube-api-access-7kwjh\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:06:21.558554 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:06:21.558521 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nv4dm" Jul 23 18:06:21.558554 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:06:21.558540 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nv4dm" event={"ID":"e030935f-a095-42e7-8caa-8de8922825cf","Type":"ContainerDied","Data":"8f7aecf61c4a4b939b54e12d9539510534497012dcd87e2ca2c52aa2e4bfd8db"} Jul 23 18:06:21.559075 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:06:21.558593 2564 scope.go:117] "RemoveContainer" containerID="1553439d9a0c2070fd30b6c6a4d2a3cbf0cfc3e935f16613db40ed36aca1781e" Jul 23 18:06:21.580577 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:06:21.580553 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nv4dm"] Jul 23 18:06:21.586240 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:06:21.586216 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nv4dm"] Jul 23 18:06:23.117920 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:06:23.117886 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e030935f-a095-42e7-8caa-8de8922825cf" path="/var/lib/kubelet/pods/e030935f-a095-42e7-8caa-8de8922825cf/volumes" Jul 23 18:07:02.665655 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:02.665616 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:07:17.504077 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:17.504044 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:07:23.203511 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:23.203476 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:07:29.360738 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:29.360703 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:07:41.494896 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.494857 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz"] Jul 23 18:07:41.500877 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.496099 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e030935f-a095-42e7-8caa-8de8922825cf" containerName="authorino" Jul 23 18:07:41.500877 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.496132 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e030935f-a095-42e7-8caa-8de8922825cf" containerName="authorino" Jul 23 18:07:41.500877 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.496280 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e030935f-a095-42e7-8caa-8de8922825cf" containerName="authorino" Jul 23 18:07:41.503291 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.503256 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.506479 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.506458 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 18:07:41.507962 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.507942 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 23 18:07:41.508610 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.508592 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 18:07:41.508863 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.508847 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jxkjt\"" Jul 23 18:07:41.510962 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.510937 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz"] Jul 23 18:07:41.573068 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.573031 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cdf622b-6f10-4378-9171-69886cfa30f3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.573240 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.573080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cdf622b-6f10-4378-9171-69886cfa30f3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.573240 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.573140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cdf622b-6f10-4378-9171-69886cfa30f3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.573240 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.573174 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28b24\" (UniqueName: \"kubernetes.io/projected/3cdf622b-6f10-4378-9171-69886cfa30f3-kube-api-access-28b24\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.573240 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.573202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cdf622b-6f10-4378-9171-69886cfa30f3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.573373 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.573279 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cdf622b-6f10-4378-9171-69886cfa30f3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.674324 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.674288 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cdf622b-6f10-4378-9171-69886cfa30f3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.674499 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.674334 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cdf622b-6f10-4378-9171-69886cfa30f3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.674499 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.674367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cdf622b-6f10-4378-9171-69886cfa30f3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.674499 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.674386 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cdf622b-6f10-4378-9171-69886cfa30f3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.674499 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.674403 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28b24\" (UniqueName: \"kubernetes.io/projected/3cdf622b-6f10-4378-9171-69886cfa30f3-kube-api-access-28b24\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.674499 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.674426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cdf622b-6f10-4378-9171-69886cfa30f3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.674762 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.674735 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cdf622b-6f10-4378-9171-69886cfa30f3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.674844 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.674763 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cdf622b-6f10-4378-9171-69886cfa30f3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.674844 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.674826 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cdf622b-6f10-4378-9171-69886cfa30f3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.676722 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.676700 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cdf622b-6f10-4378-9171-69886cfa30f3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.676846 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.676820 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cdf622b-6f10-4378-9171-69886cfa30f3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.683152 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.683133 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28b24\" (UniqueName: \"kubernetes.io/projected/3cdf622b-6f10-4378-9171-69886cfa30f3-kube-api-access-28b24\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz\" (UID: \"3cdf622b-6f10-4378-9171-69886cfa30f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.816545 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.816454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:41.948678 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:41.948652 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz"] Jul 23 18:07:42.929494 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:42.929457 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" event={"ID":"3cdf622b-6f10-4378-9171-69886cfa30f3","Type":"ContainerStarted","Data":"ebb3e7cd033049fda90cd71687807b45922f1bcd5669d10d1f04880c1b5fe53b"} Jul 23 18:07:43.004972 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:43.004950 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:07:43.935490 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:43.935451 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" event={"ID":"3cdf622b-6f10-4378-9171-69886cfa30f3","Type":"ContainerStarted","Data":"bb90a71936278d5e4cfb1c1affc66125e8ed6f8ed6fd524657645e1e7cb08f0a"} Jul 23 18:07:43.972791 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:43.972743 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" podStartSLOduration=1.936413239 podStartE2EDuration="2.972727789s" totalImagesPullingTime="1.03631455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:43.972624467 +0000 UTC m=+613.414123061" watchObservedRunningTime="2026-07-23 18:07:43.972727789 +0000 UTC m=+613.414226382" Jul 23 18:07:51.817155 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:51.817098 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:51.818609 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:51.817208 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:51.829845 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:51.829794 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:51.982723 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:51.982694 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz" Jul 23 18:07:57.305316 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:07:57.305235 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:08:08.920734 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:08.920701 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp"] Jul 23 18:08:08.923374 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:08.923358 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:08.931039 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:08.931016 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 23 18:08:08.939165 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:08.939145 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp"] Jul 23 18:08:09.006781 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.006747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25334cf6-c5e6-4aaf-b459-521146a4cc2a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.006966 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.006825 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25334cf6-c5e6-4aaf-b459-521146a4cc2a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.006966 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.006867 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25334cf6-c5e6-4aaf-b459-521146a4cc2a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.006966 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.006886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25334cf6-c5e6-4aaf-b459-521146a4cc2a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.006966 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.006928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsct9\" (UniqueName: \"kubernetes.io/projected/25334cf6-c5e6-4aaf-b459-521146a4cc2a-kube-api-access-dsct9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.007101 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.007007 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25334cf6-c5e6-4aaf-b459-521146a4cc2a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.108017 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.107962 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25334cf6-c5e6-4aaf-b459-521146a4cc2a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.108017 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.108025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25334cf6-c5e6-4aaf-b459-521146a4cc2a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.108261 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.108054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25334cf6-c5e6-4aaf-b459-521146a4cc2a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.108261 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.108077 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsct9\" (UniqueName: \"kubernetes.io/projected/25334cf6-c5e6-4aaf-b459-521146a4cc2a-kube-api-access-dsct9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.108261 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.108128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25334cf6-c5e6-4aaf-b459-521146a4cc2a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.108261 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.108162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25334cf6-c5e6-4aaf-b459-521146a4cc2a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.108447 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.108422 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25334cf6-c5e6-4aaf-b459-521146a4cc2a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.108518 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.108502 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25334cf6-c5e6-4aaf-b459-521146a4cc2a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.108576 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.108500 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25334cf6-c5e6-4aaf-b459-521146a4cc2a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.110634 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.110612 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25334cf6-c5e6-4aaf-b459-521146a4cc2a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.110743 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.110637 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25334cf6-c5e6-4aaf-b459-521146a4cc2a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.118328 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.118303 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsct9\" (UniqueName: \"kubernetes.io/projected/25334cf6-c5e6-4aaf-b459-521146a4cc2a-kube-api-access-dsct9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp\" (UID: \"25334cf6-c5e6-4aaf-b459-521146a4cc2a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.232029 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.231996 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:09.376270 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.376118 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp"] Jul 23 18:08:09.378936 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:08:09.378905 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25334cf6_c5e6_4aaf_b459_521146a4cc2a.slice/crio-e8f2582478f895f701e5cd1945e55dd8a229c81a447d35686e8f002f06851393 WatchSource:0}: Error finding container e8f2582478f895f701e5cd1945e55dd8a229c81a447d35686e8f002f06851393: Status 404 returned error can't find the container with id e8f2582478f895f701e5cd1945e55dd8a229c81a447d35686e8f002f06851393 Jul 23 18:08:09.380303 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:09.380279 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:08:10.052047 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:10.052013 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" event={"ID":"25334cf6-c5e6-4aaf-b459-521146a4cc2a","Type":"ContainerStarted","Data":"dbaac071719d2e97c5fd3748e314a6804e9f3a58578da9b0c0e3e35fc7cc8961"} Jul 23 18:08:10.052425 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:10.052057 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" event={"ID":"25334cf6-c5e6-4aaf-b459-521146a4cc2a","Type":"ContainerStarted","Data":"e8f2582478f895f701e5cd1945e55dd8a229c81a447d35686e8f002f06851393"} Jul 23 18:08:10.082605 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:10.082555 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" podStartSLOduration=1.894765383 podStartE2EDuration="2.082533943s" totalImagesPullingTime="187.76856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:08:10.079557175 +0000 UTC m=+639.521055768" watchObservedRunningTime="2026-07-23 18:08:10.082533943 +0000 UTC m=+639.524032578" Jul 23 18:08:11.315223 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:11.315187 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:08:19.232864 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:19.232830 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:19.233348 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:19.232910 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:19.245316 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:19.245295 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:20.116176 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:20.116145 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp" Jul 23 18:08:51.464586 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:51.464546 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f7c55d5c4-hphtp"] Jul 23 18:08:51.468443 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:51.468420 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f7c55d5c4-hphtp" Jul 23 18:08:51.471258 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:51.471239 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 23 18:08:51.473211 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:51.473178 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lvr5d\"" Jul 23 18:08:51.478965 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:51.478939 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f7c55d5c4-hphtp"] Jul 23 18:08:51.576510 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:51.576471 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d1abcb93-9bdb-4800-9ed5-b2817085798a-tls-cert\") pod \"authorino-f7c55d5c4-hphtp\" (UID: \"d1abcb93-9bdb-4800-9ed5-b2817085798a\") " pod="kuadrant-system/authorino-f7c55d5c4-hphtp" Jul 23 18:08:51.576710 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:51.576581 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7qpv\" (UniqueName: \"kubernetes.io/projected/d1abcb93-9bdb-4800-9ed5-b2817085798a-kube-api-access-g7qpv\") pod \"authorino-f7c55d5c4-hphtp\" (UID: \"d1abcb93-9bdb-4800-9ed5-b2817085798a\") " pod="kuadrant-system/authorino-f7c55d5c4-hphtp" Jul 23 18:08:51.677222 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:51.677188 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qpv\" (UniqueName: \"kubernetes.io/projected/d1abcb93-9bdb-4800-9ed5-b2817085798a-kube-api-access-g7qpv\") pod \"authorino-f7c55d5c4-hphtp\" (UID: \"d1abcb93-9bdb-4800-9ed5-b2817085798a\") " pod="kuadrant-system/authorino-f7c55d5c4-hphtp" Jul 23 18:08:51.677444 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:51.677293 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d1abcb93-9bdb-4800-9ed5-b2817085798a-tls-cert\") pod \"authorino-f7c55d5c4-hphtp\" (UID: \"d1abcb93-9bdb-4800-9ed5-b2817085798a\") " pod="kuadrant-system/authorino-f7c55d5c4-hphtp" Jul 23 18:08:51.679737 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:51.679712 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d1abcb93-9bdb-4800-9ed5-b2817085798a-tls-cert\") pod \"authorino-f7c55d5c4-hphtp\" (UID: \"d1abcb93-9bdb-4800-9ed5-b2817085798a\") " pod="kuadrant-system/authorino-f7c55d5c4-hphtp" Jul 23 18:08:51.690383 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:51.690360 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7qpv\" (UniqueName: \"kubernetes.io/projected/d1abcb93-9bdb-4800-9ed5-b2817085798a-kube-api-access-g7qpv\") pod \"authorino-f7c55d5c4-hphtp\" (UID: \"d1abcb93-9bdb-4800-9ed5-b2817085798a\") " pod="kuadrant-system/authorino-f7c55d5c4-hphtp" Jul 23 18:08:51.778925 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:51.778893 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f7c55d5c4-hphtp" Jul 23 18:08:51.924508 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:51.924411 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f7c55d5c4-hphtp"] Jul 23 18:08:51.926776 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:08:51.926744 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1abcb93_9bdb_4800_9ed5_b2817085798a.slice/crio-86fc247643e17922503cd5029e8d223312755334fc39fc9cf0c0e5da1bff37ff WatchSource:0}: Error finding container 86fc247643e17922503cd5029e8d223312755334fc39fc9cf0c0e5da1bff37ff: Status 404 returned error can't find the container with id 86fc247643e17922503cd5029e8d223312755334fc39fc9cf0c0e5da1bff37ff Jul 23 18:08:52.240192 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:52.240157 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f7c55d5c4-hphtp" event={"ID":"d1abcb93-9bdb-4800-9ed5-b2817085798a","Type":"ContainerStarted","Data":"86fc247643e17922503cd5029e8d223312755334fc39fc9cf0c0e5da1bff37ff"} Jul 23 18:08:53.249358 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:53.249317 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f7c55d5c4-hphtp" event={"ID":"d1abcb93-9bdb-4800-9ed5-b2817085798a","Type":"ContainerStarted","Data":"5ce29238e94d85ca4b72a6e959cfd60a400e814ffbbc95bf2f67e4718570bf37"} Jul 23 18:08:53.296948 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:08:53.296897 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f7c55d5c4-hphtp" podStartSLOduration=1.8559389880000001 podStartE2EDuration="2.29688218s" totalImagesPullingTime="440.943192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:08:53.296094647 +0000 UTC m=+682.737593264" watchObservedRunningTime="2026-07-23 18:08:53.29688218 +0000 UTC m=+682.738380773" Jul 23 18:09:21.316944 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:09:21.316908 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:09:31.707522 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:09:31.707438 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:09:42.401674 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:09:42.401639 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:11:24.581333 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:11:24.581285 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:12:00.946494 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:12:00.946456 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:12:10.613991 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:12:10.613958 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:12:19.812573 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:12:19.812538 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:12:30.534701 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:12:30.534620 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:12:39.935741 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:12:39.935707 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:12:51.722412 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:12:51.722373 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:13:03.204285 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:13:03.204245 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:13:13.101062 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:13:13.101024 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:13:38.396107 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:13:38.396074 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:13:50.995882 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:13:50.995844 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:14:13.894883 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:14:13.894852 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:14:24.993008 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:14:24.992964 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:14:41.289730 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:14:41.289692 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:14:50.689643 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:14:50.689603 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:15:00.389778 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:15:00.389732 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:15:14.394425 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:15:14.394391 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:17:09.291441 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:17:09.291366 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:17:20.789552 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:17:20.789511 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:17:29.785824 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:17:29.781901 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:17:39.592007 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:17:39.591970 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:17:49.684906 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:17:49.684872 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:17:57.391524 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:17:57.391491 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:18:08.485675 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:18:08.485641 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:18:18.487475 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:18:18.487440 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:18:29.189096 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:18:29.189022 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:18:38.487847 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:18:38.487793 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:18:46.495344 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:18:46.495311 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:18:58.597819 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:18:58.597774 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:19:20.684565 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:20.684527 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:19:30.487339 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:30.487303 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:19:45.896521 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:45.896490 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:19:56.392269 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.392187 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8"] Jul 23 18:19:56.395732 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.395714 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.398302 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.398284 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-cc577869-kserve-self-signed-certs\"" Jul 23 18:19:56.406558 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.406532 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8"] Jul 23 18:19:56.487337 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.487300 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-home\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.487505 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.487409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-dshm\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.487505 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.487431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-model-cache\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.487505 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.487454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35bec46e-75b3-4576-8e53-2f083c7353f7-tls-certs\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.487505 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.487502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-tmp-dir\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.487640 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.487520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6dtp\" (UniqueName: \"kubernetes.io/projected/35bec46e-75b3-4576-8e53-2f083c7353f7-kube-api-access-w6dtp\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.588441 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.588405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-model-cache\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.588441 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.588442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35bec46e-75b3-4576-8e53-2f083c7353f7-tls-certs\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.588640 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.588483 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-tmp-dir\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.588640 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.588511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dtp\" (UniqueName: \"kubernetes.io/projected/35bec46e-75b3-4576-8e53-2f083c7353f7-kube-api-access-w6dtp\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.588640 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.588552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-home\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.588760 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.588638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-dshm\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.588976 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.588952 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-tmp-dir\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.589052 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.589002 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-home\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.589052 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.589002 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-model-cache\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.590839 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.590794 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-dshm\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.591003 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.590986 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35bec46e-75b3-4576-8e53-2f083c7353f7-tls-certs\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.596575 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.596553 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6dtp\" (UniqueName: \"kubernetes.io/projected/35bec46e-75b3-4576-8e53-2f083c7353f7-kube-api-access-w6dtp\") pod \"e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.705567 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.705467 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:19:56.841551 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:56.841527 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8"] Jul 23 18:19:56.843800 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:19:56.843764 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35bec46e_75b3_4576_8e53_2f083c7353f7.slice/crio-ef1d4863e0fcb8e2857b43ee6f467294e6f109d5f039b3104e484216d6d16cc9 WatchSource:0}: Error finding container ef1d4863e0fcb8e2857b43ee6f467294e6f109d5f039b3104e484216d6d16cc9: Status 404 returned error can't find the container with id ef1d4863e0fcb8e2857b43ee6f467294e6f109d5f039b3104e484216d6d16cc9 Jul 23 18:19:57.360188 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:57.360150 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" event={"ID":"35bec46e-75b3-4576-8e53-2f083c7353f7","Type":"ContainerStarted","Data":"d5441b94ebdf04bb0feba323ea44a0052f889f4cd2bd890570c1821d93072355"} Jul 23 18:19:57.360188 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:57.360196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" event={"ID":"35bec46e-75b3-4576-8e53-2f083c7353f7","Type":"ContainerStarted","Data":"ef1d4863e0fcb8e2857b43ee6f467294e6f109d5f039b3104e484216d6d16cc9"} Jul 23 18:19:57.379001 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:19:57.378953 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" podStartSLOduration=1.378937625 podStartE2EDuration="1.378937625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:19:57.37754909 +0000 UTC m=+1346.819047718" watchObservedRunningTime="2026-07-23 18:19:57.378937625 +0000 UTC m=+1346.820436231" Jul 23 18:20:00.491262 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:00.491220 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:20:06.705876 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:06.705789 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:20:06.705876 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:06.705892 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:20:06.718700 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:06.718677 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:20:07.419074 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:07.419047 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:20:27.766405 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:27.766369 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8"] Jul 23 18:20:30.387658 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:30.387614 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:20:42.786658 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:42.786585 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" podUID="35bec46e-75b3-4576-8e53-2f083c7353f7" containerName="main" containerID="cri-o://d5441b94ebdf04bb0feba323ea44a0052f889f4cd2bd890570c1821d93072355" gracePeriod=45 Jul 23 18:20:43.034677 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.034653 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:20:43.089393 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.089316 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35bec46e-75b3-4576-8e53-2f083c7353f7-tls-certs\") pod \"35bec46e-75b3-4576-8e53-2f083c7353f7\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " Jul 23 18:20:43.089393 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.089358 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6dtp\" (UniqueName: \"kubernetes.io/projected/35bec46e-75b3-4576-8e53-2f083c7353f7-kube-api-access-w6dtp\") pod \"35bec46e-75b3-4576-8e53-2f083c7353f7\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " Jul 23 18:20:43.089393 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.089391 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-tmp-dir\") pod \"35bec46e-75b3-4576-8e53-2f083c7353f7\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " Jul 23 18:20:43.089652 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.089568 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-home\") pod \"35bec46e-75b3-4576-8e53-2f083c7353f7\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " Jul 23 18:20:43.089713 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.089648 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-model-cache\") pod \"35bec46e-75b3-4576-8e53-2f083c7353f7\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " Jul 23 18:20:43.089713 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.089661 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35bec46e-75b3-4576-8e53-2f083c7353f7" (UID: "35bec46e-75b3-4576-8e53-2f083c7353f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:20:43.089713 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.089696 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-dshm\") pod \"35bec46e-75b3-4576-8e53-2f083c7353f7\" (UID: \"35bec46e-75b3-4576-8e53-2f083c7353f7\") " Jul 23 18:20:43.089892 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.089828 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-home" (OuterVolumeSpecName: "home") pod "35bec46e-75b3-4576-8e53-2f083c7353f7" (UID: "35bec46e-75b3-4576-8e53-2f083c7353f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:20:43.089945 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.089895 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "35bec46e-75b3-4576-8e53-2f083c7353f7" (UID: "35bec46e-75b3-4576-8e53-2f083c7353f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:20:43.090023 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.090008 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-model-cache\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:20:43.090058 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.090028 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-tmp-dir\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:20:43.090058 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.090036 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-home\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:20:43.091738 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.091716 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-dshm" (OuterVolumeSpecName: "dshm") pod "35bec46e-75b3-4576-8e53-2f083c7353f7" (UID: "35bec46e-75b3-4576-8e53-2f083c7353f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:20:43.092037 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.092020 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bec46e-75b3-4576-8e53-2f083c7353f7-kube-api-access-w6dtp" (OuterVolumeSpecName: "kube-api-access-w6dtp") pod "35bec46e-75b3-4576-8e53-2f083c7353f7" (UID: "35bec46e-75b3-4576-8e53-2f083c7353f7"). InnerVolumeSpecName "kube-api-access-w6dtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:20:43.092099 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.092031 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35bec46e-75b3-4576-8e53-2f083c7353f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35bec46e-75b3-4576-8e53-2f083c7353f7" (UID: "35bec46e-75b3-4576-8e53-2f083c7353f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:20:43.191435 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.191393 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35bec46e-75b3-4576-8e53-2f083c7353f7-tls-certs\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:20:43.191435 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.191430 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6dtp\" (UniqueName: \"kubernetes.io/projected/35bec46e-75b3-4576-8e53-2f083c7353f7-kube-api-access-w6dtp\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:20:43.191435 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.191443 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35bec46e-75b3-4576-8e53-2f083c7353f7-dshm\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:20:43.581139 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.581100 2564 generic.go:356] "Generic (PLEG): container finished" podID="35bec46e-75b3-4576-8e53-2f083c7353f7" containerID="d5441b94ebdf04bb0feba323ea44a0052f889f4cd2bd890570c1821d93072355" exitCode=0 Jul 23 18:20:43.581310 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.581143 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" Jul 23 18:20:43.581310 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.581158 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" event={"ID":"35bec46e-75b3-4576-8e53-2f083c7353f7","Type":"ContainerDied","Data":"d5441b94ebdf04bb0feba323ea44a0052f889f4cd2bd890570c1821d93072355"} Jul 23 18:20:43.581310 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.581199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8" event={"ID":"35bec46e-75b3-4576-8e53-2f083c7353f7","Type":"ContainerDied","Data":"ef1d4863e0fcb8e2857b43ee6f467294e6f109d5f039b3104e484216d6d16cc9"} Jul 23 18:20:43.581310 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.581216 2564 scope.go:117] "RemoveContainer" containerID="d5441b94ebdf04bb0feba323ea44a0052f889f4cd2bd890570c1821d93072355" Jul 23 18:20:43.590007 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.589980 2564 scope.go:117] "RemoveContainer" containerID="d5441b94ebdf04bb0feba323ea44a0052f889f4cd2bd890570c1821d93072355" Jul 23 18:20:43.590229 ip-10-0-142-147 kubenswrapper[2564]: E0723 18:20:43.590214 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5441b94ebdf04bb0feba323ea44a0052f889f4cd2bd890570c1821d93072355\": container with ID starting with d5441b94ebdf04bb0feba323ea44a0052f889f4cd2bd890570c1821d93072355 not found: ID does not exist" containerID="d5441b94ebdf04bb0feba323ea44a0052f889f4cd2bd890570c1821d93072355" Jul 23 18:20:43.590281 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.590244 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5441b94ebdf04bb0feba323ea44a0052f889f4cd2bd890570c1821d93072355"} err="failed to get container status \"d5441b94ebdf04bb0feba323ea44a0052f889f4cd2bd890570c1821d93072355\": rpc error: code = NotFound desc = could not find container \"d5441b94ebdf04bb0feba323ea44a0052f889f4cd2bd890570c1821d93072355\": container with ID starting with d5441b94ebdf04bb0feba323ea44a0052f889f4cd2bd890570c1821d93072355 not found: ID does not exist" Jul 23 18:20:43.598090 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.598066 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8"] Jul 23 18:20:43.602294 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.602273 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-cc577869-kserve-6cd6c5cddc-kxsb8"] Jul 23 18:20:43.983202 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:43.983175 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:20:45.117459 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:45.117426 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35bec46e-75b3-4576-8e53-2f083c7353f7" path="/var/lib/kubelet/pods/35bec46e-75b3-4576-8e53-2f083c7353f7/volumes" Jul 23 18:20:52.787401 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:20:52.787367 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:21:07.383370 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:21:07.383334 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:21:17.195988 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:21:17.195952 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:21:26.686035 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:21:26.685950 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:21:36.790772 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:21:36.790734 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:21:45.386854 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:21:45.386817 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:21:55.091647 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:21:55.091604 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:22:08.190343 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:22:08.190304 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:22:19.087270 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:22:19.087236 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:22:28.496893 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:22:28.496835 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:22:40.594106 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:22:40.594075 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:22:53.003385 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:22:53.003346 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:23:02.096869 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:23:02.096827 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:23:13.790516 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:23:13.790481 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:23:25.206465 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:23:25.206425 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:23:36.284772 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:23:36.284737 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:23:49.187165 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:23:49.187122 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:23:58.587781 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:23:58.587737 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:24:08.687323 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:24:08.687284 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:24:19.783070 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:24:19.783032 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:24:52.394347 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:24:52.394268 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:25:03.388345 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:25:03.388310 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:25:36.095584 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:25:36.095548 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:25:47.295885 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:25:47.295853 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:26:20.595036 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:26:20.594999 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:26:30.379746 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:26:30.379703 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:27:05.494993 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:27:05.494960 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:28:36.611318 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:28:36.611282 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:28:45.594564 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:28:45.594520 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:29:01.200722 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:29:01.200638 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:29:20.200858 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:29:20.200819 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:29:56.392188 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:29:56.392151 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:30:12.489278 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:30:12.489240 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:30:26.199404 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:30:26.199320 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:30:44.385303 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:30:44.385262 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:30:52.886582 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:30:52.886541 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:31:59.506828 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:31:59.506716 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:32:09.713234 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:32:09.713179 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:33:02.191820 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:02.191770 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:33:18.197654 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:18.197612 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:33:19.202385 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.202355 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg"] Jul 23 18:33:19.202759 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.202709 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35bec46e-75b3-4576-8e53-2f083c7353f7" containerName="main" Jul 23 18:33:19.202759 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.202720 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="35bec46e-75b3-4576-8e53-2f083c7353f7" containerName="main" Jul 23 18:33:19.202850 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.202776 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="35bec46e-75b3-4576-8e53-2f083c7353f7" containerName="main" Jul 23 18:33:19.205952 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.205935 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.209785 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.209760 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-954ecd\"/\"test-model-954ecd-kserve-self-signed-certs\"" Jul 23 18:33:19.209967 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.209760 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-954ecd\"/\"openshift-service-ca.crt\"" Jul 23 18:33:19.209967 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.209848 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-954ecd\"/\"kube-root-ca.crt\"" Jul 23 18:33:19.209967 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.209914 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-954ecd\"/\"default-dockercfg-6zw6r\"" Jul 23 18:33:19.219327 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.219307 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg"] Jul 23 18:33:19.284411 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.284373 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6deec783-d984-4a43-8444-5d2389374dda-tls-certs\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.284617 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.284436 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-model-cache\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.284617 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.284465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-home\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.284617 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.284495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-tmp-dir\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.284617 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.284528 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrbd\" (UniqueName: \"kubernetes.io/projected/6deec783-d984-4a43-8444-5d2389374dda-kube-api-access-mhrbd\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.284617 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.284578 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-dshm\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.385749 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.385722 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrbd\" (UniqueName: \"kubernetes.io/projected/6deec783-d984-4a43-8444-5d2389374dda-kube-api-access-mhrbd\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.385924 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.385772 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-dshm\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.385924 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.385828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6deec783-d984-4a43-8444-5d2389374dda-tls-certs\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.385924 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.385875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-model-cache\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.385924 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.385892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-home\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.385924 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.385915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-tmp-dir\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.386306 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.386284 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-home\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.386306 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.386299 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-model-cache\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.386408 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.386357 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-tmp-dir\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.387999 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.387978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-dshm\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.388259 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.388242 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6deec783-d984-4a43-8444-5d2389374dda-tls-certs\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.394782 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.394759 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrbd\" (UniqueName: \"kubernetes.io/projected/6deec783-d984-4a43-8444-5d2389374dda-kube-api-access-mhrbd\") pod \"test-model-954ecd-kserve-576d9b46bf-4fzjg\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.515783 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.515725 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:19.651324 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:19.651300 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg"] Jul 23 18:33:19.653339 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:33:19.653311 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6deec783_d984_4a43_8444_5d2389374dda.slice/crio-28153e9951c17579d6b95182658515829c2fe29aa11e544b93e93090ba9c3078 WatchSource:0}: Error finding container 28153e9951c17579d6b95182658515829c2fe29aa11e544b93e93090ba9c3078: Status 404 returned error can't find the container with id 28153e9951c17579d6b95182658515829c2fe29aa11e544b93e93090ba9c3078 Jul 23 18:33:20.053400 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:20.053366 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" event={"ID":"6deec783-d984-4a43-8444-5d2389374dda","Type":"ContainerStarted","Data":"cb5a697765a9bcaf36a3e89e03f8e0b2a2f537d3264c1b02c449cda0a4f0a5ba"} Jul 23 18:33:20.053400 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:20.053405 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" event={"ID":"6deec783-d984-4a43-8444-5d2389374dda","Type":"ContainerStarted","Data":"28153e9951c17579d6b95182658515829c2fe29aa11e544b93e93090ba9c3078"} Jul 23 18:33:20.071697 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:20.071647 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" podStartSLOduration=1.071631931 podStartE2EDuration="1.071631931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:33:20.071359974 +0000 UTC m=+2149.512858569" watchObservedRunningTime="2026-07-23 18:33:20.071631931 +0000 UTC m=+2149.513130525" Jul 23 18:33:29.516137 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:29.516033 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:29.516137 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:29.516096 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:29.528523 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:29.528494 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:30.114012 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:30.113980 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:33:30.196597 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:30.196564 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:33:45.997434 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:45.997395 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:33:59.793291 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:33:59.793251 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:34:23.944011 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:23.943976 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg"] Jul 23 18:34:28.594879 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:28.594845 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:34:38.964111 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:38.964048 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" podUID="6deec783-d984-4a43-8444-5d2389374dda" containerName="main" containerID="cri-o://cb5a697765a9bcaf36a3e89e03f8e0b2a2f537d3264c1b02c449cda0a4f0a5ba" gracePeriod=45 Jul 23 18:34:39.211129 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.211103 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:34:39.295600 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.295561 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-home\") pod \"6deec783-d984-4a43-8444-5d2389374dda\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " Jul 23 18:34:39.295600 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.295607 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6deec783-d984-4a43-8444-5d2389374dda-tls-certs\") pod \"6deec783-d984-4a43-8444-5d2389374dda\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " Jul 23 18:34:39.295866 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.295659 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhrbd\" (UniqueName: \"kubernetes.io/projected/6deec783-d984-4a43-8444-5d2389374dda-kube-api-access-mhrbd\") pod \"6deec783-d984-4a43-8444-5d2389374dda\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " Jul 23 18:34:39.295866 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.295684 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-tmp-dir\") pod \"6deec783-d984-4a43-8444-5d2389374dda\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " Jul 23 18:34:39.295866 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.295735 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-dshm\") pod \"6deec783-d984-4a43-8444-5d2389374dda\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " Jul 23 18:34:39.295866 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.295843 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-model-cache\") pod \"6deec783-d984-4a43-8444-5d2389374dda\" (UID: \"6deec783-d984-4a43-8444-5d2389374dda\") " Jul 23 18:34:39.296083 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.295893 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-home" (OuterVolumeSpecName: "home") pod "6deec783-d984-4a43-8444-5d2389374dda" (UID: "6deec783-d984-4a43-8444-5d2389374dda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:34:39.296083 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.296017 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6deec783-d984-4a43-8444-5d2389374dda" (UID: "6deec783-d984-4a43-8444-5d2389374dda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:34:39.296204 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.296087 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-tmp-dir\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:34:39.296204 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.296101 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-home\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:34:39.296204 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.296113 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-model-cache" (OuterVolumeSpecName: "model-cache") pod "6deec783-d984-4a43-8444-5d2389374dda" (UID: "6deec783-d984-4a43-8444-5d2389374dda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:34:39.298068 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.298048 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6deec783-d984-4a43-8444-5d2389374dda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6deec783-d984-4a43-8444-5d2389374dda" (UID: "6deec783-d984-4a43-8444-5d2389374dda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:34:39.298126 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.298088 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6deec783-d984-4a43-8444-5d2389374dda-kube-api-access-mhrbd" (OuterVolumeSpecName: "kube-api-access-mhrbd") pod "6deec783-d984-4a43-8444-5d2389374dda" (UID: "6deec783-d984-4a43-8444-5d2389374dda"). InnerVolumeSpecName "kube-api-access-mhrbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:34:39.298126 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.298111 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-dshm" (OuterVolumeSpecName: "dshm") pod "6deec783-d984-4a43-8444-5d2389374dda" (UID: "6deec783-d984-4a43-8444-5d2389374dda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:34:39.397201 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.397166 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6deec783-d984-4a43-8444-5d2389374dda-tls-certs\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:34:39.397201 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.397195 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrbd\" (UniqueName: \"kubernetes.io/projected/6deec783-d984-4a43-8444-5d2389374dda-kube-api-access-mhrbd\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:34:39.397201 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.397205 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-dshm\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:34:39.397436 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.397216 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6deec783-d984-4a43-8444-5d2389374dda-model-cache\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:34:39.425553 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.425519 2564 generic.go:356] "Generic (PLEG): container finished" podID="6deec783-d984-4a43-8444-5d2389374dda" containerID="cb5a697765a9bcaf36a3e89e03f8e0b2a2f537d3264c1b02c449cda0a4f0a5ba" exitCode=0 Jul 23 18:34:39.425708 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.425556 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" Jul 23 18:34:39.425708 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.425603 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" event={"ID":"6deec783-d984-4a43-8444-5d2389374dda","Type":"ContainerDied","Data":"cb5a697765a9bcaf36a3e89e03f8e0b2a2f537d3264c1b02c449cda0a4f0a5ba"} Jul 23 18:34:39.425708 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.425642 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg" event={"ID":"6deec783-d984-4a43-8444-5d2389374dda","Type":"ContainerDied","Data":"28153e9951c17579d6b95182658515829c2fe29aa11e544b93e93090ba9c3078"} Jul 23 18:34:39.425708 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.425660 2564 scope.go:117] "RemoveContainer" containerID="cb5a697765a9bcaf36a3e89e03f8e0b2a2f537d3264c1b02c449cda0a4f0a5ba" Jul 23 18:34:39.434598 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.434579 2564 scope.go:117] "RemoveContainer" containerID="cb5a697765a9bcaf36a3e89e03f8e0b2a2f537d3264c1b02c449cda0a4f0a5ba" Jul 23 18:34:39.434855 ip-10-0-142-147 kubenswrapper[2564]: E0723 18:34:39.434834 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5a697765a9bcaf36a3e89e03f8e0b2a2f537d3264c1b02c449cda0a4f0a5ba\": container with ID starting with cb5a697765a9bcaf36a3e89e03f8e0b2a2f537d3264c1b02c449cda0a4f0a5ba not found: ID does not exist" containerID="cb5a697765a9bcaf36a3e89e03f8e0b2a2f537d3264c1b02c449cda0a4f0a5ba" Jul 23 18:34:39.434960 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.434863 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5a697765a9bcaf36a3e89e03f8e0b2a2f537d3264c1b02c449cda0a4f0a5ba"} err="failed to get container status \"cb5a697765a9bcaf36a3e89e03f8e0b2a2f537d3264c1b02c449cda0a4f0a5ba\": rpc error: code = NotFound desc = could not find container \"cb5a697765a9bcaf36a3e89e03f8e0b2a2f537d3264c1b02c449cda0a4f0a5ba\": container with ID starting with cb5a697765a9bcaf36a3e89e03f8e0b2a2f537d3264c1b02c449cda0a4f0a5ba not found: ID does not exist" Jul 23 18:34:39.458822 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.458780 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg"] Jul 23 18:34:39.459517 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:39.459261 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-954ecd/test-model-954ecd-kserve-576d9b46bf-4fzjg"] Jul 23 18:34:41.117542 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:41.117507 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6deec783-d984-4a43-8444-5d2389374dda" path="/var/lib/kubelet/pods/6deec783-d984-4a43-8444-5d2389374dda/volumes" Jul 23 18:34:44.092103 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:34:44.092069 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:35:00.895824 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:35:00.895720 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:35:21.288344 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:35:21.288308 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:35:38.297933 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:35:38.297893 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:35:54.192751 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:35:54.192714 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:36:21.890048 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:36:21.890013 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:36:40.691297 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:36:40.691217 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:36:54.698333 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:36:54.698292 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:37:09.195597 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:09.195565 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:37:55.633306 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.633214 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp"] Jul 23 18:37:55.634770 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.634747 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6deec783-d984-4a43-8444-5d2389374dda" containerName="main" Jul 23 18:37:55.634934 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.634922 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6deec783-d984-4a43-8444-5d2389374dda" containerName="main" Jul 23 18:37:55.635248 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.635232 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6deec783-d984-4a43-8444-5d2389374dda" containerName="main" Jul 23 18:37:55.638684 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.638665 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.641829 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.641792 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f5044c\"/\"default-dockercfg-z9t5b\"" Jul 23 18:37:55.641919 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.641794 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f5044c\"/\"openshift-service-ca.crt\"" Jul 23 18:37:55.642851 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.642838 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f5044c\"/\"kube-root-ca.crt\"" Jul 23 18:37:55.642906 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.642897 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f5044c\"/\"auth-test-model-f5044c-kserve-self-signed-certs\"" Jul 23 18:37:55.654698 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.654674 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp"] Jul 23 18:37:55.793797 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.793762 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f8f150e-cf73-472f-9842-f4052e6c488d-tls-certs\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.793976 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.793899 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-model-cache\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.793976 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.793928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-tmp-dir\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.793976 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.793959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-dshm\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.794080 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.793985 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2kw6\" (UniqueName: \"kubernetes.io/projected/5f8f150e-cf73-472f-9842-f4052e6c488d-kube-api-access-s2kw6\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.794080 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.794060 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-home\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.895283 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.895200 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-dshm\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.895283 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.895240 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2kw6\" (UniqueName: \"kubernetes.io/projected/5f8f150e-cf73-472f-9842-f4052e6c488d-kube-api-access-s2kw6\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.895283 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.895271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-home\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.895562 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.895292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f8f150e-cf73-472f-9842-f4052e6c488d-tls-certs\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.895562 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.895341 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-model-cache\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.895562 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.895356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-tmp-dir\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.895722 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.895669 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-home\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.895722 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.895679 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-tmp-dir\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.895841 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.895761 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-model-cache\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.897593 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.897572 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-dshm\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.897971 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.897950 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f8f150e-cf73-472f-9842-f4052e6c488d-tls-certs\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.906773 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.906750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2kw6\" (UniqueName: \"kubernetes.io/projected/5f8f150e-cf73-472f-9842-f4052e6c488d-kube-api-access-s2kw6\") pod \"auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:55.947620 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:55.947587 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:37:56.084491 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:56.084461 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp"] Jul 23 18:37:56.087555 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:37:56.087531 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f8f150e_cf73_472f_9842_f4052e6c488d.slice/crio-27898db4462d1be4b1a9e52d0b68a24c21db7dc319640da4e418e12c82d4f62d WatchSource:0}: Error finding container 27898db4462d1be4b1a9e52d0b68a24c21db7dc319640da4e418e12c82d4f62d: Status 404 returned error can't find the container with id 27898db4462d1be4b1a9e52d0b68a24c21db7dc319640da4e418e12c82d4f62d Jul 23 18:37:56.327446 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:56.327407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" event={"ID":"5f8f150e-cf73-472f-9842-f4052e6c488d","Type":"ContainerStarted","Data":"acd8af077cdbb3ddeda8e09f9943b9b8fc1cef5078f70fef96c379f586c89d63"} Jul 23 18:37:56.327446 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:56.327454 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" event={"ID":"5f8f150e-cf73-472f-9842-f4052e6c488d","Type":"ContainerStarted","Data":"27898db4462d1be4b1a9e52d0b68a24c21db7dc319640da4e418e12c82d4f62d"} Jul 23 18:37:56.357161 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:56.357087 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" podStartSLOduration=1.357070803 podStartE2EDuration="1.357070803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:37:56.35650988 +0000 UTC m=+2425.798008475" watchObservedRunningTime="2026-07-23 18:37:56.357070803 +0000 UTC m=+2425.798569396" Jul 23 18:37:59.919785 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:37:59.919742 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:38:05.948364 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:38:05.948330 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:38:05.948364 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:38:05.948371 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:38:05.960979 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:38:05.960956 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:38:06.387282 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:38:06.387255 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:38:14.217604 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:38:14.217561 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:38:29.799631 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:38:29.799599 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:38:48.900254 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:38:48.900222 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:39:09.356073 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:39:09.356041 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:39:27.731311 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:39:27.731228 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:39:42.999818 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:39:42.999768 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:40:00.428864 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:00.428824 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp"] Jul 23 18:40:01.207611 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:01.207574 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:40:02.818224 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:02.818190 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84"] Jul 23 18:40:02.821656 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:02.821637 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:02.825887 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:02.825861 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e94d59\"/\"openshift-service-ca.crt\"" Jul 23 18:40:02.825887 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:02.825883 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e94d59\"/\"kube-root-ca.crt\"" Jul 23 18:40:02.827660 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:02.827640 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e94d59\"/\"default-dockercfg-6mnfs\"" Jul 23 18:40:02.827786 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:02.827765 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e94d59\"/\"sub-test-model-e94d59-kserve-self-signed-certs\"" Jul 23 18:40:02.847675 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:02.847644 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84"] Jul 23 18:40:02.928685 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:02.928650 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-tls-certs\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:02.928904 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:02.928702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-home\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:02.928904 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:02.928722 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-dshm\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:02.928904 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:02.928766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-tmp-dir\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:02.928904 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:02.928863 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m72m4\" (UniqueName: \"kubernetes.io/projected/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-kube-api-access-m72m4\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:02.929070 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:02.928906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-model-cache\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:03.029985 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.029931 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-home\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:03.029985 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.029982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-dshm\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:03.030204 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.030095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-tmp-dir\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:03.030204 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.030186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m72m4\" (UniqueName: \"kubernetes.io/projected/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-kube-api-access-m72m4\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:03.030302 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.030251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-model-cache\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:03.030353 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.030303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-tls-certs\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:03.030402 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.030373 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-home\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:03.030487 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.030464 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-tmp-dir\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:03.030597 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.030583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-model-cache\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:03.032273 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.032247 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-dshm\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:03.032593 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.032577 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-tls-certs\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:03.042715 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.042686 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m72m4\" (UniqueName: \"kubernetes.io/projected/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-kube-api-access-m72m4\") pod \"sub-test-model-e94d59-kserve-756fcf67f9-k8c84\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:03.131249 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.131160 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:03.271035 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.270997 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84"] Jul 23 18:40:03.919651 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.919608 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" event={"ID":"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0","Type":"ContainerStarted","Data":"50e290569f9c86e9f50c5cea0556b5987dca60406ddb698901bf9589e16f8320"} Jul 23 18:40:03.920100 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.919660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" event={"ID":"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0","Type":"ContainerStarted","Data":"2c5e46d1ba64bf7ca57283541c498d6573b9ea77aa3f7273b1512835dceea5ae"} Jul 23 18:40:03.942059 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:03.942010 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" podStartSLOduration=1.941993318 podStartE2EDuration="1.941993318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:40:03.941618201 +0000 UTC m=+2553.383116798" watchObservedRunningTime="2026-07-23 18:40:03.941993318 +0000 UTC m=+2553.383491911" Jul 23 18:40:12.402921 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:12.402889 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:40:13.132041 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:13.132004 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:13.132252 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:13.132055 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:13.144605 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:13.144577 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:13.977917 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:13.977888 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:40:15.449328 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.449292 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" podUID="5f8f150e-cf73-472f-9842-f4052e6c488d" containerName="main" containerID="cri-o://acd8af077cdbb3ddeda8e09f9943b9b8fc1cef5078f70fef96c379f586c89d63" gracePeriod=45 Jul 23 18:40:15.702954 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.702895 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:40:15.849640 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.849601 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-dshm\") pod \"5f8f150e-cf73-472f-9842-f4052e6c488d\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " Jul 23 18:40:15.849640 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.849637 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-home\") pod \"5f8f150e-cf73-472f-9842-f4052e6c488d\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " Jul 23 18:40:15.849873 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.849659 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-model-cache\") pod \"5f8f150e-cf73-472f-9842-f4052e6c488d\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " Jul 23 18:40:15.849873 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.849686 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2kw6\" (UniqueName: \"kubernetes.io/projected/5f8f150e-cf73-472f-9842-f4052e6c488d-kube-api-access-s2kw6\") pod \"5f8f150e-cf73-472f-9842-f4052e6c488d\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " Jul 23 18:40:15.849873 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.849765 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-tmp-dir\") pod \"5f8f150e-cf73-472f-9842-f4052e6c488d\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " Jul 23 18:40:15.849873 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.849848 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f8f150e-cf73-472f-9842-f4052e6c488d-tls-certs\") pod \"5f8f150e-cf73-472f-9842-f4052e6c488d\" (UID: \"5f8f150e-cf73-472f-9842-f4052e6c488d\") " Jul 23 18:40:15.850053 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.849981 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f8f150e-cf73-472f-9842-f4052e6c488d" (UID: "5f8f150e-cf73-472f-9842-f4052e6c488d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:15.850053 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.849995 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-home" (OuterVolumeSpecName: "home") pod "5f8f150e-cf73-472f-9842-f4052e6c488d" (UID: "5f8f150e-cf73-472f-9842-f4052e6c488d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:15.850161 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.850078 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f8f150e-cf73-472f-9842-f4052e6c488d" (UID: "5f8f150e-cf73-472f-9842-f4052e6c488d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:15.850218 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.850186 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-home\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:40:15.850218 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.850202 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-model-cache\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:40:15.850218 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.850211 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-tmp-dir\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:40:15.852044 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.852010 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-dshm" (OuterVolumeSpecName: "dshm") pod "5f8f150e-cf73-472f-9842-f4052e6c488d" (UID: "5f8f150e-cf73-472f-9842-f4052e6c488d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:15.852158 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.852079 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8f150e-cf73-472f-9842-f4052e6c488d-kube-api-access-s2kw6" (OuterVolumeSpecName: "kube-api-access-s2kw6") pod "5f8f150e-cf73-472f-9842-f4052e6c488d" (UID: "5f8f150e-cf73-472f-9842-f4052e6c488d"). InnerVolumeSpecName "kube-api-access-s2kw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:40:15.852158 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.852094 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f8f150e-cf73-472f-9842-f4052e6c488d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f8f150e-cf73-472f-9842-f4052e6c488d" (UID: "5f8f150e-cf73-472f-9842-f4052e6c488d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:40:15.950709 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.950673 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f8f150e-cf73-472f-9842-f4052e6c488d-dshm\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:40:15.950709 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.950703 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2kw6\" (UniqueName: \"kubernetes.io/projected/5f8f150e-cf73-472f-9842-f4052e6c488d-kube-api-access-s2kw6\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:40:15.950709 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.950717 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f8f150e-cf73-472f-9842-f4052e6c488d-tls-certs\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:40:15.977039 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.977001 2564 generic.go:356] "Generic (PLEG): container finished" podID="5f8f150e-cf73-472f-9842-f4052e6c488d" containerID="acd8af077cdbb3ddeda8e09f9943b9b8fc1cef5078f70fef96c379f586c89d63" exitCode=0 Jul 23 18:40:15.977197 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.977045 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" Jul 23 18:40:15.977197 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.977045 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" event={"ID":"5f8f150e-cf73-472f-9842-f4052e6c488d","Type":"ContainerDied","Data":"acd8af077cdbb3ddeda8e09f9943b9b8fc1cef5078f70fef96c379f586c89d63"} Jul 23 18:40:15.977197 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.977143 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp" event={"ID":"5f8f150e-cf73-472f-9842-f4052e6c488d","Type":"ContainerDied","Data":"27898db4462d1be4b1a9e52d0b68a24c21db7dc319640da4e418e12c82d4f62d"} Jul 23 18:40:15.977197 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.977164 2564 scope.go:117] "RemoveContainer" containerID="acd8af077cdbb3ddeda8e09f9943b9b8fc1cef5078f70fef96c379f586c89d63" Jul 23 18:40:15.985865 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.985848 2564 scope.go:117] "RemoveContainer" containerID="acd8af077cdbb3ddeda8e09f9943b9b8fc1cef5078f70fef96c379f586c89d63" Jul 23 18:40:15.986137 ip-10-0-142-147 kubenswrapper[2564]: E0723 18:40:15.986120 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd8af077cdbb3ddeda8e09f9943b9b8fc1cef5078f70fef96c379f586c89d63\": container with ID starting with acd8af077cdbb3ddeda8e09f9943b9b8fc1cef5078f70fef96c379f586c89d63 not found: ID does not exist" containerID="acd8af077cdbb3ddeda8e09f9943b9b8fc1cef5078f70fef96c379f586c89d63" Jul 23 18:40:15.986203 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:15.986148 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd8af077cdbb3ddeda8e09f9943b9b8fc1cef5078f70fef96c379f586c89d63"} err="failed to get container status \"acd8af077cdbb3ddeda8e09f9943b9b8fc1cef5078f70fef96c379f586c89d63\": rpc error: code = NotFound desc = could not find container \"acd8af077cdbb3ddeda8e09f9943b9b8fc1cef5078f70fef96c379f586c89d63\": container with ID starting with acd8af077cdbb3ddeda8e09f9943b9b8fc1cef5078f70fef96c379f586c89d63 not found: ID does not exist" Jul 23 18:40:16.000236 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:16.000213 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp"] Jul 23 18:40:16.005440 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:16.005416 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f5044c/auth-test-model-f5044c-kserve-7b6499cbc5-dbvsp"] Jul 23 18:40:17.117437 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:17.117402 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f8f150e-cf73-472f-9842-f4052e6c488d" path="/var/lib/kubelet/pods/5f8f150e-cf73-472f-9842-f4052e6c488d/volumes" Jul 23 18:40:23.495344 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:23.495307 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:40:38.305639 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:38.305606 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:40:54.197529 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:40:54.197444 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:41:00.441907 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:00.441871 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84"] Jul 23 18:41:10.416175 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.416142 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64"] Jul 23 18:41:10.416624 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.416511 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f8f150e-cf73-472f-9842-f4052e6c488d" containerName="main" Jul 23 18:41:10.416624 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.416521 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f8f150e-cf73-472f-9842-f4052e6c488d" containerName="main" Jul 23 18:41:10.416624 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.416589 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f8f150e-cf73-472f-9842-f4052e6c488d" containerName="main" Jul 23 18:41:10.419487 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.419470 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.424403 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.424382 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e94d59\"/\"rate-test-model-e94d59-kserve-self-signed-certs\"" Jul 23 18:41:10.434678 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.434648 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64"] Jul 23 18:41:10.537974 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.537940 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-home\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.538146 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.537983 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-model-cache\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.538146 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.538006 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-tmp-dir\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.538146 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.538086 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7eefe84-e338-4395-b825-6bac0a91f1e1-tls-certs\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.538146 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.538131 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsfsl\" (UniqueName: \"kubernetes.io/projected/a7eefe84-e338-4395-b825-6bac0a91f1e1-kube-api-access-nsfsl\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.538286 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.538156 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-dshm\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.638655 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.638619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-model-cache\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.638655 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.638654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-tmp-dir\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.638938 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.638678 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7eefe84-e338-4395-b825-6bac0a91f1e1-tls-certs\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.638938 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.638705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsfsl\" (UniqueName: \"kubernetes.io/projected/a7eefe84-e338-4395-b825-6bac0a91f1e1-kube-api-access-nsfsl\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.638938 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.638733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-dshm\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.638938 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.638845 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-home\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.639146 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.639096 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-model-cache\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.639249 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.639231 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-tmp-dir\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.639303 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.639250 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-home\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.641066 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.641045 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-dshm\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.641491 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.641471 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7eefe84-e338-4395-b825-6bac0a91f1e1-tls-certs\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.649602 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.649581 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsfsl\" (UniqueName: \"kubernetes.io/projected/a7eefe84-e338-4395-b825-6bac0a91f1e1-kube-api-access-nsfsl\") pod \"rate-test-model-e94d59-kserve-6f5cfd478c-q6h64\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.728763 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.728724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:10.869165 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:10.869136 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64"] Jul 23 18:41:10.871897 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:41:10.871861 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7eefe84_e338_4395_b825_6bac0a91f1e1.slice/crio-a2aae8f84b175e6993f439e04c67d55a6e665f2ffc6d7aaa3fafa4024279ad10 WatchSource:0}: Error finding container a2aae8f84b175e6993f439e04c67d55a6e665f2ffc6d7aaa3fafa4024279ad10: Status 404 returned error can't find the container with id a2aae8f84b175e6993f439e04c67d55a6e665f2ffc6d7aaa3fafa4024279ad10 Jul 23 18:41:11.241331 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:11.241294 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" event={"ID":"a7eefe84-e338-4395-b825-6bac0a91f1e1","Type":"ContainerStarted","Data":"e80f58a405bae6e9a037aa10ddf00287131a46333586d19922cc2eb62430ba09"} Jul 23 18:41:11.241331 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:11.241337 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" event={"ID":"a7eefe84-e338-4395-b825-6bac0a91f1e1","Type":"ContainerStarted","Data":"a2aae8f84b175e6993f439e04c67d55a6e665f2ffc6d7aaa3fafa4024279ad10"} Jul 23 18:41:11.265592 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:11.265547 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" podStartSLOduration=1.265533051 podStartE2EDuration="1.265533051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:41:11.264091888 +0000 UTC m=+2620.705590483" watchObservedRunningTime="2026-07-23 18:41:11.265533051 +0000 UTC m=+2620.707031645" Jul 23 18:41:13.306197 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:13.306161 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:41:15.461902 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.461828 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" podUID="2e1acfdf-1c9b-4a85-8e6f-71f535eccba0" containerName="main" containerID="cri-o://50e290569f9c86e9f50c5cea0556b5987dca60406ddb698901bf9589e16f8320" gracePeriod=45 Jul 23 18:41:15.708401 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.708375 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:41:15.886693 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.886654 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-model-cache\") pod \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " Jul 23 18:41:15.886876 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.886727 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-tmp-dir\") pod \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " Jul 23 18:41:15.886876 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.886748 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m72m4\" (UniqueName: \"kubernetes.io/projected/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-kube-api-access-m72m4\") pod \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " Jul 23 18:41:15.886876 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.886772 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-tls-certs\") pod \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " Jul 23 18:41:15.886876 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.886799 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-home\") pod \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " Jul 23 18:41:15.886876 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.886870 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-dshm\") pod \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\" (UID: \"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0\") " Jul 23 18:41:15.887147 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.886992 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e1acfdf-1c9b-4a85-8e6f-71f535eccba0" (UID: "2e1acfdf-1c9b-4a85-8e6f-71f535eccba0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:41:15.887147 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.887052 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e1acfdf-1c9b-4a85-8e6f-71f535eccba0" (UID: "2e1acfdf-1c9b-4a85-8e6f-71f535eccba0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:41:15.887239 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.887158 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-home" (OuterVolumeSpecName: "home") pod "2e1acfdf-1c9b-4a85-8e6f-71f535eccba0" (UID: "2e1acfdf-1c9b-4a85-8e6f-71f535eccba0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:41:15.887239 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.887186 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-model-cache\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:41:15.887239 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.887203 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-tmp-dir\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:41:15.889124 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.889093 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e1acfdf-1c9b-4a85-8e6f-71f535eccba0" (UID: "2e1acfdf-1c9b-4a85-8e6f-71f535eccba0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:41:15.889241 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.889136 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-kube-api-access-m72m4" (OuterVolumeSpecName: "kube-api-access-m72m4") pod "2e1acfdf-1c9b-4a85-8e6f-71f535eccba0" (UID: "2e1acfdf-1c9b-4a85-8e6f-71f535eccba0"). InnerVolumeSpecName "kube-api-access-m72m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:41:15.889241 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.889153 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-dshm" (OuterVolumeSpecName: "dshm") pod "2e1acfdf-1c9b-4a85-8e6f-71f535eccba0" (UID: "2e1acfdf-1c9b-4a85-8e6f-71f535eccba0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:41:15.988611 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.988567 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m72m4\" (UniqueName: \"kubernetes.io/projected/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-kube-api-access-m72m4\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:41:15.988611 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.988604 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-tls-certs\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:41:15.988611 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.988618 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-home\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:41:15.988893 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:15.988627 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0-dshm\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:41:16.266204 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:16.266168 2564 generic.go:356] "Generic (PLEG): container finished" podID="2e1acfdf-1c9b-4a85-8e6f-71f535eccba0" containerID="50e290569f9c86e9f50c5cea0556b5987dca60406ddb698901bf9589e16f8320" exitCode=0 Jul 23 18:41:16.266411 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:16.266212 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" Jul 23 18:41:16.266411 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:16.266259 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" event={"ID":"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0","Type":"ContainerDied","Data":"50e290569f9c86e9f50c5cea0556b5987dca60406ddb698901bf9589e16f8320"} Jul 23 18:41:16.266411 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:16.266304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84" event={"ID":"2e1acfdf-1c9b-4a85-8e6f-71f535eccba0","Type":"ContainerDied","Data":"2c5e46d1ba64bf7ca57283541c498d6573b9ea77aa3f7273b1512835dceea5ae"} Jul 23 18:41:16.266411 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:16.266328 2564 scope.go:117] "RemoveContainer" containerID="50e290569f9c86e9f50c5cea0556b5987dca60406ddb698901bf9589e16f8320" Jul 23 18:41:16.277515 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:16.277498 2564 scope.go:117] "RemoveContainer" containerID="50e290569f9c86e9f50c5cea0556b5987dca60406ddb698901bf9589e16f8320" Jul 23 18:41:16.277764 ip-10-0-142-147 kubenswrapper[2564]: E0723 18:41:16.277746 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e290569f9c86e9f50c5cea0556b5987dca60406ddb698901bf9589e16f8320\": container with ID starting with 50e290569f9c86e9f50c5cea0556b5987dca60406ddb698901bf9589e16f8320 not found: ID does not exist" containerID="50e290569f9c86e9f50c5cea0556b5987dca60406ddb698901bf9589e16f8320" Jul 23 18:41:16.277895 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:16.277775 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e290569f9c86e9f50c5cea0556b5987dca60406ddb698901bf9589e16f8320"} err="failed to get container status \"50e290569f9c86e9f50c5cea0556b5987dca60406ddb698901bf9589e16f8320\": rpc error: code = NotFound desc = could not find container \"50e290569f9c86e9f50c5cea0556b5987dca60406ddb698901bf9589e16f8320\": container with ID starting with 50e290569f9c86e9f50c5cea0556b5987dca60406ddb698901bf9589e16f8320 not found: ID does not exist" Jul 23 18:41:16.292641 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:16.292610 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84"] Jul 23 18:41:16.297554 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:16.297528 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e94d59/sub-test-model-e94d59-kserve-756fcf67f9-k8c84"] Jul 23 18:41:17.117282 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:17.117251 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e1acfdf-1c9b-4a85-8e6f-71f535eccba0" path="/var/lib/kubelet/pods/2e1acfdf-1c9b-4a85-8e6f-71f535eccba0/volumes" Jul 23 18:41:20.729529 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:20.729493 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:20.729950 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:20.729545 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:20.741849 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:20.741827 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:21.302309 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:21.302282 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:41:43.203979 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:41:43.203943 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:42:13.530402 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:13.530366 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:42:28.944920 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:28.944833 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:42:36.751224 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:36.751188 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64"] Jul 23 18:42:41.413651 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:41.413609 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7585-140210901893168"] Jul 23 18:42:41.414188 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:41.414151 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1acfdf-1c9b-4a85-8e6f-71f535eccba0" containerName="main" Jul 23 18:42:41.414188 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:41.414170 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1acfdf-1c9b-4a85-8e6f-71f535eccba0" containerName="main" Jul 23 18:42:41.414310 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:41.414297 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e1acfdf-1c9b-4a85-8e6f-71f535eccba0" containerName="main" Jul 23 18:42:41.417334 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:41.417314 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-140210901893168" Jul 23 18:42:41.420412 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:41.420386 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mr4c9\"" Jul 23 18:42:41.422997 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:41.422974 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7585-140210901893168"] Jul 23 18:42:41.527149 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:41.527112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnzlx\" (UniqueName: \"kubernetes.io/projected/f196bf1d-4d4d-4682-9a78-1ddc6880a7bd-kube-api-access-gnzlx\") pod \"test-curl-7585-140210901893168\" (UID: \"f196bf1d-4d4d-4682-9a78-1ddc6880a7bd\") " pod="opendatahub/test-curl-7585-140210901893168" Jul 23 18:42:41.628544 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:41.628496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnzlx\" (UniqueName: \"kubernetes.io/projected/f196bf1d-4d4d-4682-9a78-1ddc6880a7bd-kube-api-access-gnzlx\") pod \"test-curl-7585-140210901893168\" (UID: \"f196bf1d-4d4d-4682-9a78-1ddc6880a7bd\") " pod="opendatahub/test-curl-7585-140210901893168" Jul 23 18:42:41.637791 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:41.637757 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnzlx\" (UniqueName: \"kubernetes.io/projected/f196bf1d-4d4d-4682-9a78-1ddc6880a7bd-kube-api-access-gnzlx\") pod \"test-curl-7585-140210901893168\" (UID: \"f196bf1d-4d4d-4682-9a78-1ddc6880a7bd\") " pod="opendatahub/test-curl-7585-140210901893168" Jul 23 18:42:41.727462 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:41.727433 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-140210901893168" Jul 23 18:42:42.068423 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:42.068387 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7585-140210901893168"] Jul 23 18:42:42.070661 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:42:42.070634 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf196bf1d_4d4d_4682_9a78_1ddc6880a7bd.slice/crio-24e0ed0f48d2e92e51500c33f24a9705854198a0dfcf176ca0c17879eca20b09 WatchSource:0}: Error finding container 24e0ed0f48d2e92e51500c33f24a9705854198a0dfcf176ca0c17879eca20b09: Status 404 returned error can't find the container with id 24e0ed0f48d2e92e51500c33f24a9705854198a0dfcf176ca0c17879eca20b09 Jul 23 18:42:42.071799 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:42.071774 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:42:42.667980 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:42.667944 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7585-140210901893168" event={"ID":"f196bf1d-4d4d-4682-9a78-1ddc6880a7bd","Type":"ContainerStarted","Data":"24e0ed0f48d2e92e51500c33f24a9705854198a0dfcf176ca0c17879eca20b09"} Jul 23 18:42:43.591273 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:43.591237 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:42:43.673485 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:43.673448 2564 generic.go:356] "Generic (PLEG): container finished" podID="f196bf1d-4d4d-4682-9a78-1ddc6880a7bd" containerID="5f1b200cf09d6ecf8bb60636121576f3ed1e5f64582e422fca72244b5e85b836" exitCode=0 Jul 23 18:42:43.673940 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:43.673521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7585-140210901893168" event={"ID":"f196bf1d-4d4d-4682-9a78-1ddc6880a7bd","Type":"ContainerDied","Data":"5f1b200cf09d6ecf8bb60636121576f3ed1e5f64582e422fca72244b5e85b836"} Jul 23 18:42:44.812955 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:44.812924 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-140210901893168" Jul 23 18:42:44.853533 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:44.853500 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7585-140210901893168_f196bf1d-4d4d-4682-9a78-1ddc6880a7bd/test-curl-7585-140210901893168/0.log" Jul 23 18:42:44.854696 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:44.854665 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnzlx\" (UniqueName: \"kubernetes.io/projected/f196bf1d-4d4d-4682-9a78-1ddc6880a7bd-kube-api-access-gnzlx\") pod \"f196bf1d-4d4d-4682-9a78-1ddc6880a7bd\" (UID: \"f196bf1d-4d4d-4682-9a78-1ddc6880a7bd\") " Jul 23 18:42:44.856795 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:44.856770 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f196bf1d-4d4d-4682-9a78-1ddc6880a7bd-kube-api-access-gnzlx" (OuterVolumeSpecName: "kube-api-access-gnzlx") pod "f196bf1d-4d4d-4682-9a78-1ddc6880a7bd" (UID: "f196bf1d-4d4d-4682-9a78-1ddc6880a7bd"). InnerVolumeSpecName "kube-api-access-gnzlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:42:44.878918 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:44.878891 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7585-140210901893168"] Jul 23 18:42:44.883076 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:44.883055 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7585-140210901893168"] Jul 23 18:42:44.955670 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:44.955590 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnzlx\" (UniqueName: \"kubernetes.io/projected/f196bf1d-4d4d-4682-9a78-1ddc6880a7bd-kube-api-access-gnzlx\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:42:45.117391 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:45.117344 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f196bf1d-4d4d-4682-9a78-1ddc6880a7bd" path="/var/lib/kubelet/pods/f196bf1d-4d4d-4682-9a78-1ddc6880a7bd/volumes" Jul 23 18:42:45.684427 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:45.684398 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-140210901893168" Jul 23 18:42:45.684599 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:45.684399 2564 scope.go:117] "RemoveContainer" containerID="5f1b200cf09d6ecf8bb60636121576f3ed1e5f64582e422fca72244b5e85b836" Jul 23 18:42:47.435284 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:47.435248 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7585-140210904409408"] Jul 23 18:42:47.435714 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:47.435617 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f196bf1d-4d4d-4682-9a78-1ddc6880a7bd" containerName="test-curl-7585-140210901893168" Jul 23 18:42:47.435714 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:47.435629 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f196bf1d-4d4d-4682-9a78-1ddc6880a7bd" containerName="test-curl-7585-140210901893168" Jul 23 18:42:47.435714 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:47.435684 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f196bf1d-4d4d-4682-9a78-1ddc6880a7bd" containerName="test-curl-7585-140210901893168" Jul 23 18:42:47.439980 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:47.439965 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-140210904409408" Jul 23 18:42:47.444449 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:47.444296 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mr4c9\"" Jul 23 18:42:47.445135 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:47.445108 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7585-140210904409408"] Jul 23 18:42:47.580078 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:47.580030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwcx7\" (UniqueName: \"kubernetes.io/projected/d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209-kube-api-access-pwcx7\") pod \"test-curl-7585-140210904409408\" (UID: \"d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209\") " pod="opendatahub/test-curl-7585-140210904409408" Jul 23 18:42:47.680538 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:47.680482 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwcx7\" (UniqueName: \"kubernetes.io/projected/d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209-kube-api-access-pwcx7\") pod \"test-curl-7585-140210904409408\" (UID: \"d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209\") " pod="opendatahub/test-curl-7585-140210904409408" Jul 23 18:42:47.689929 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:47.689866 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwcx7\" (UniqueName: \"kubernetes.io/projected/d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209-kube-api-access-pwcx7\") pod \"test-curl-7585-140210904409408\" (UID: \"d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209\") " pod="opendatahub/test-curl-7585-140210904409408" Jul 23 18:42:47.751477 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:47.751417 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-140210904409408" Jul 23 18:42:47.885031 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:47.885006 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7585-140210904409408"] Jul 23 18:42:47.886891 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:42:47.886860 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5e0fed8_31e9_4959_9c4e_1d7fc6f1e209.slice/crio-4b4c5cfc5af3dc39c99faeb61501551c649971cb803025655a7814b7ff5654a4 WatchSource:0}: Error finding container 4b4c5cfc5af3dc39c99faeb61501551c649971cb803025655a7814b7ff5654a4: Status 404 returned error can't find the container with id 4b4c5cfc5af3dc39c99faeb61501551c649971cb803025655a7814b7ff5654a4 Jul 23 18:42:48.700611 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:48.700580 2564 generic.go:356] "Generic (PLEG): container finished" podID="d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209" containerID="f1646b76d4c268231a5d59cee69c800d25517c5fcff9b6da6e1ab552690e1cb0" exitCode=0 Jul 23 18:42:48.700948 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:48.700660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7585-140210904409408" event={"ID":"d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209","Type":"ContainerDied","Data":"f1646b76d4c268231a5d59cee69c800d25517c5fcff9b6da6e1ab552690e1cb0"} Jul 23 18:42:48.700948 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:48.700701 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7585-140210904409408" event={"ID":"d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209","Type":"ContainerStarted","Data":"4b4c5cfc5af3dc39c99faeb61501551c649971cb803025655a7814b7ff5654a4"} Jul 23 18:42:49.833865 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:49.833843 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-140210904409408" Jul 23 18:42:49.861318 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:49.861288 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7585-140210904409408_d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209/test-curl-7585-140210904409408/0.log" Jul 23 18:42:49.889717 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:49.889694 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7585-140210904409408"] Jul 23 18:42:49.897566 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:49.897544 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwcx7\" (UniqueName: \"kubernetes.io/projected/d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209-kube-api-access-pwcx7\") pod \"d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209\" (UID: \"d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209\") " Jul 23 18:42:49.897971 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:49.897953 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7585-140210904409408"] Jul 23 18:42:49.899701 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:49.899674 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209-kube-api-access-pwcx7" (OuterVolumeSpecName: "kube-api-access-pwcx7") pod "d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209" (UID: "d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209"). InnerVolumeSpecName "kube-api-access-pwcx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:42:49.998396 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:49.998364 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwcx7\" (UniqueName: \"kubernetes.io/projected/d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209-kube-api-access-pwcx7\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:42:50.710876 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:50.710839 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-140210904409408" Jul 23 18:42:50.711027 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:50.710842 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b4c5cfc5af3dc39c99faeb61501551c649971cb803025655a7814b7ff5654a4" Jul 23 18:42:50.721879 ip-10-0-142-147 kubenswrapper[2564]: E0723 18:42:50.721848 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7585-140210904409408\" is forbidden: User \"system:node:ip-10-0-142-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-147.ec2.internal' and this object" podUID="d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209" pod="opendatahub/test-curl-7585-140210904409408" Jul 23 18:42:51.116821 ip-10-0-142-147 kubenswrapper[2564]: E0723 18:42:51.116779 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7585-140210904409408\" is forbidden: User \"system:node:ip-10-0-142-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-147.ec2.internal' and this object" podUID="d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209" pod="opendatahub/test-curl-7585-140210904409408" Jul 23 18:42:51.117859 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:51.117834 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209" path="/var/lib/kubelet/pods/d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209/volumes" Jul 23 18:42:51.770831 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:51.770777 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" podUID="a7eefe84-e338-4395-b825-6bac0a91f1e1" containerName="main" containerID="cri-o://e80f58a405bae6e9a037aa10ddf00287131a46333586d19922cc2eb62430ba09" gracePeriod=45 Jul 23 18:42:52.022215 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.022150 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:42:52.116277 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.116244 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7eefe84-e338-4395-b825-6bac0a91f1e1-tls-certs\") pod \"a7eefe84-e338-4395-b825-6bac0a91f1e1\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " Jul 23 18:42:52.116432 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.116288 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsfsl\" (UniqueName: \"kubernetes.io/projected/a7eefe84-e338-4395-b825-6bac0a91f1e1-kube-api-access-nsfsl\") pod \"a7eefe84-e338-4395-b825-6bac0a91f1e1\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " Jul 23 18:42:52.116432 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.116330 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-home\") pod \"a7eefe84-e338-4395-b825-6bac0a91f1e1\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " Jul 23 18:42:52.116432 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.116354 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-dshm\") pod \"a7eefe84-e338-4395-b825-6bac0a91f1e1\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " Jul 23 18:42:52.116569 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.116456 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-tmp-dir\") pod \"a7eefe84-e338-4395-b825-6bac0a91f1e1\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " Jul 23 18:42:52.116569 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.116485 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-model-cache\") pod \"a7eefe84-e338-4395-b825-6bac0a91f1e1\" (UID: \"a7eefe84-e338-4395-b825-6bac0a91f1e1\") " Jul 23 18:42:52.116680 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.116639 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-home" (OuterVolumeSpecName: "home") pod "a7eefe84-e338-4395-b825-6bac0a91f1e1" (UID: "a7eefe84-e338-4395-b825-6bac0a91f1e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:42:52.116771 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.116741 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7eefe84-e338-4395-b825-6bac0a91f1e1" (UID: "a7eefe84-e338-4395-b825-6bac0a91f1e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:42:52.116867 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.116839 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-home\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:42:52.117172 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.116899 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7eefe84-e338-4395-b825-6bac0a91f1e1" (UID: "a7eefe84-e338-4395-b825-6bac0a91f1e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:42:52.118492 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.118461 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7eefe84-e338-4395-b825-6bac0a91f1e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7eefe84-e338-4395-b825-6bac0a91f1e1" (UID: "a7eefe84-e338-4395-b825-6bac0a91f1e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:42:52.119063 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.119041 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-dshm" (OuterVolumeSpecName: "dshm") pod "a7eefe84-e338-4395-b825-6bac0a91f1e1" (UID: "a7eefe84-e338-4395-b825-6bac0a91f1e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:42:52.119141 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.119064 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7eefe84-e338-4395-b825-6bac0a91f1e1-kube-api-access-nsfsl" (OuterVolumeSpecName: "kube-api-access-nsfsl") pod "a7eefe84-e338-4395-b825-6bac0a91f1e1" (UID: "a7eefe84-e338-4395-b825-6bac0a91f1e1"). InnerVolumeSpecName "kube-api-access-nsfsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:42:52.218490 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.218442 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-tmp-dir\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:42:52.218490 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.218485 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-model-cache\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:42:52.218490 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.218498 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7eefe84-e338-4395-b825-6bac0a91f1e1-tls-certs\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:42:52.218840 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.218512 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsfsl\" (UniqueName: \"kubernetes.io/projected/a7eefe84-e338-4395-b825-6bac0a91f1e1-kube-api-access-nsfsl\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:42:52.218840 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.218525 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7eefe84-e338-4395-b825-6bac0a91f1e1-dshm\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:42:52.475550 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.475517 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7585-140210904409120"] Jul 23 18:42:52.475940 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.475926 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7eefe84-e338-4395-b825-6bac0a91f1e1" containerName="main" Jul 23 18:42:52.475993 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.475944 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7eefe84-e338-4395-b825-6bac0a91f1e1" containerName="main" Jul 23 18:42:52.475993 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.475962 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209" containerName="test-curl-7585-140210904409408" Jul 23 18:42:52.475993 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.475968 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209" containerName="test-curl-7585-140210904409408" Jul 23 18:42:52.476145 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.476042 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7eefe84-e338-4395-b825-6bac0a91f1e1" containerName="main" Jul 23 18:42:52.476145 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.476052 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5e0fed8-31e9-4959-9c4e-1d7fc6f1e209" containerName="test-curl-7585-140210904409408" Jul 23 18:42:52.479165 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.479149 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-140210904409120" Jul 23 18:42:52.482917 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.482896 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mr4c9\"" Jul 23 18:42:52.487653 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.487629 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7585-140210904409120"] Jul 23 18:42:52.622160 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.622121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntbbn\" (UniqueName: \"kubernetes.io/projected/f255df26-62ae-4e34-8b0f-f743bcf583bf-kube-api-access-ntbbn\") pod \"test-curl-7585-140210904409120\" (UID: \"f255df26-62ae-4e34-8b0f-f743bcf583bf\") " pod="opendatahub/test-curl-7585-140210904409120" Jul 23 18:42:52.721178 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.721141 2564 generic.go:356] "Generic (PLEG): container finished" podID="a7eefe84-e338-4395-b825-6bac0a91f1e1" containerID="e80f58a405bae6e9a037aa10ddf00287131a46333586d19922cc2eb62430ba09" exitCode=0 Jul 23 18:42:52.721356 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.721206 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" Jul 23 18:42:52.721356 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.721214 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" event={"ID":"a7eefe84-e338-4395-b825-6bac0a91f1e1","Type":"ContainerDied","Data":"e80f58a405bae6e9a037aa10ddf00287131a46333586d19922cc2eb62430ba09"} Jul 23 18:42:52.721356 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.721261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64" event={"ID":"a7eefe84-e338-4395-b825-6bac0a91f1e1","Type":"ContainerDied","Data":"a2aae8f84b175e6993f439e04c67d55a6e665f2ffc6d7aaa3fafa4024279ad10"} Jul 23 18:42:52.721356 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.721281 2564 scope.go:117] "RemoveContainer" containerID="e80f58a405bae6e9a037aa10ddf00287131a46333586d19922cc2eb62430ba09" Jul 23 18:42:52.722608 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.722588 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntbbn\" (UniqueName: \"kubernetes.io/projected/f255df26-62ae-4e34-8b0f-f743bcf583bf-kube-api-access-ntbbn\") pod \"test-curl-7585-140210904409120\" (UID: \"f255df26-62ae-4e34-8b0f-f743bcf583bf\") " pod="opendatahub/test-curl-7585-140210904409120" Jul 23 18:42:52.732060 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.732041 2564 scope.go:117] "RemoveContainer" containerID="e80f58a405bae6e9a037aa10ddf00287131a46333586d19922cc2eb62430ba09" Jul 23 18:42:52.732282 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.732261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntbbn\" (UniqueName: \"kubernetes.io/projected/f255df26-62ae-4e34-8b0f-f743bcf583bf-kube-api-access-ntbbn\") pod \"test-curl-7585-140210904409120\" (UID: \"f255df26-62ae-4e34-8b0f-f743bcf583bf\") " pod="opendatahub/test-curl-7585-140210904409120" Jul 23 18:42:52.732353 ip-10-0-142-147 kubenswrapper[2564]: E0723 18:42:52.732318 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80f58a405bae6e9a037aa10ddf00287131a46333586d19922cc2eb62430ba09\": container with ID starting with e80f58a405bae6e9a037aa10ddf00287131a46333586d19922cc2eb62430ba09 not found: ID does not exist" containerID="e80f58a405bae6e9a037aa10ddf00287131a46333586d19922cc2eb62430ba09" Jul 23 18:42:52.732353 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.732339 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80f58a405bae6e9a037aa10ddf00287131a46333586d19922cc2eb62430ba09"} err="failed to get container status \"e80f58a405bae6e9a037aa10ddf00287131a46333586d19922cc2eb62430ba09\": rpc error: code = NotFound desc = could not find container \"e80f58a405bae6e9a037aa10ddf00287131a46333586d19922cc2eb62430ba09\": container with ID starting with e80f58a405bae6e9a037aa10ddf00287131a46333586d19922cc2eb62430ba09 not found: ID does not exist" Jul 23 18:42:52.744615 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.744587 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64"] Jul 23 18:42:52.751325 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.751299 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e94d59/rate-test-model-e94d59-kserve-6f5cfd478c-q6h64"] Jul 23 18:42:52.788780 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.788749 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-140210904409120" Jul 23 18:42:52.923506 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:52.923477 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7585-140210904409120"] Jul 23 18:42:52.925929 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:42:52.925900 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf255df26_62ae_4e34_8b0f_f743bcf583bf.slice/crio-162631b11a5be569d8ba89f21e65f0e3aff1cc9c72442cd00b7a558d6422b408 WatchSource:0}: Error finding container 162631b11a5be569d8ba89f21e65f0e3aff1cc9c72442cd00b7a558d6422b408: Status 404 returned error can't find the container with id 162631b11a5be569d8ba89f21e65f0e3aff1cc9c72442cd00b7a558d6422b408 Jul 23 18:42:53.117944 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:53.117868 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7eefe84-e338-4395-b825-6bac0a91f1e1" path="/var/lib/kubelet/pods/a7eefe84-e338-4395-b825-6bac0a91f1e1/volumes" Jul 23 18:42:53.728616 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:53.728580 2564 generic.go:356] "Generic (PLEG): container finished" podID="f255df26-62ae-4e34-8b0f-f743bcf583bf" containerID="f432fc17e44d5e2cfee02c52db4df937f2cc9e5acd9b60b925b55a55a6eea5b8" exitCode=0 Jul 23 18:42:53.728846 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:53.728624 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7585-140210904409120" event={"ID":"f255df26-62ae-4e34-8b0f-f743bcf583bf","Type":"ContainerDied","Data":"f432fc17e44d5e2cfee02c52db4df937f2cc9e5acd9b60b925b55a55a6eea5b8"} Jul 23 18:42:53.728846 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:53.728649 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7585-140210904409120" event={"ID":"f255df26-62ae-4e34-8b0f-f743bcf583bf","Type":"ContainerStarted","Data":"162631b11a5be569d8ba89f21e65f0e3aff1cc9c72442cd00b7a558d6422b408"} Jul 23 18:42:54.869703 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:54.869672 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-140210904409120" Jul 23 18:42:54.896179 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:54.896149 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7585-140210904409120_f255df26-62ae-4e34-8b0f-f743bcf583bf/test-curl-7585-140210904409120/0.log" Jul 23 18:42:54.921283 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:54.921251 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7585-140210904409120"] Jul 23 18:42:54.924338 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:54.924304 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7585-140210904409120"] Jul 23 18:42:54.940964 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:54.940937 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntbbn\" (UniqueName: \"kubernetes.io/projected/f255df26-62ae-4e34-8b0f-f743bcf583bf-kube-api-access-ntbbn\") pod \"f255df26-62ae-4e34-8b0f-f743bcf583bf\" (UID: \"f255df26-62ae-4e34-8b0f-f743bcf583bf\") " Jul 23 18:42:54.942963 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:54.942940 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f255df26-62ae-4e34-8b0f-f743bcf583bf-kube-api-access-ntbbn" (OuterVolumeSpecName: "kube-api-access-ntbbn") pod "f255df26-62ae-4e34-8b0f-f743bcf583bf" (UID: "f255df26-62ae-4e34-8b0f-f743bcf583bf"). InnerVolumeSpecName "kube-api-access-ntbbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:42:55.042608 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:55.042512 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntbbn\" (UniqueName: \"kubernetes.io/projected/f255df26-62ae-4e34-8b0f-f743bcf583bf-kube-api-access-ntbbn\") on node \"ip-10-0-142-147.ec2.internal\" DevicePath \"\"" Jul 23 18:42:55.118076 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:55.118043 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f255df26-62ae-4e34-8b0f-f743bcf583bf" path="/var/lib/kubelet/pods/f255df26-62ae-4e34-8b0f-f743bcf583bf/volumes" Jul 23 18:42:55.739234 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:55.739203 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-140210904409120" Jul 23 18:42:55.739426 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:42:55.739254 2564 scope.go:117] "RemoveContainer" containerID="f432fc17e44d5e2cfee02c52db4df937f2cc9e5acd9b60b925b55a55a6eea5b8" Jul 23 18:43:08.596053 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:43:08.596016 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:43:31.095276 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:43:31.095236 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2vcp"] Jul 23 18:43:40.691703 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:43:40.691667 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f7c55d5c4-hphtp_d1abcb93-9bdb-4800-9ed5-b2817085798a/authorino/0.log" Jul 23 18:43:47.279546 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:43:47.279520 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f47b6774-kb9x5_79ee29b5-8549-413b-af02-2621002d17e2/copy-manifests/0.log" Jul 23 18:43:47.287493 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:43:47.287466 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f47b6774-kb9x5_79ee29b5-8549-413b-af02-2621002d17e2/manager/0.log" Jul 23 18:43:49.422265 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:43:49.422237 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f7c55d5c4-hphtp_d1abcb93-9bdb-4800-9ed5-b2817085798a/authorino/0.log" Jul 23 18:43:49.792267 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:43:49.792237 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-f4j27_2f969d3d-c3ed-45b8-b651-cfd4a64bb13a/kuadrant-console-plugin/0.log" Jul 23 18:43:50.162756 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:43:50.162675 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-m2vcp_3463c88d-e270-434f-9e29-c5b6fe4ac468/limitador/0.log" Jul 23 18:43:50.904609 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:43:50.904578 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cdff8959d-5s8vm_961f48be-e367-4efd-bbeb-570bf06a5d68/kube-auth-proxy/0.log" Jul 23 18:43:51.511439 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:43:51.511408 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-944979c7d-2pgzv_1fbd69ae-ab96-406e-97c8-d36b6c314b83/router/0.log" Jul 23 18:43:52.224077 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:43:52.224043 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhxcz_3cdf622b-6f10-4378-9171-69886cfa30f3/main/0.log" Jul 23 18:43:52.465214 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:43:52.465188 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-bjkdp_25334cf6-c5e6-4aaf-b459-521146a4cc2a/main/0.log" Jul 23 18:44:02.093945 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:02.093860 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-865dp_1d669e26-a0e5-4073-8f2c-64b52d9f959d/global-pull-secret-syncer/0.log" Jul 23 18:44:02.417313 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:02.417235 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9rmwf_e2c7a4c9-5c1f-45df-a280-ed8a43b90a93/konnectivity-agent/0.log" Jul 23 18:44:02.678663 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:02.678577 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-147.ec2.internal_4a44435e6f3df5d50b47d277e77112d1/haproxy/0.log" Jul 23 18:44:07.593517 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:07.593483 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f7c55d5c4-hphtp_d1abcb93-9bdb-4800-9ed5-b2817085798a/authorino/0.log" Jul 23 18:44:07.725222 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:07.725196 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-f4j27_2f969d3d-c3ed-45b8-b651-cfd4a64bb13a/kuadrant-console-plugin/0.log" Jul 23 18:44:08.054428 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:08.054390 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-m2vcp_3463c88d-e270-434f-9e29-c5b6fe4ac468/limitador/0.log" Jul 23 18:44:10.090089 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:10.090038 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-m6zz7_6e4ab588-dddb-4af3-a430-a4722529b3b2/kube-state-metrics/0.log" Jul 23 18:44:10.117902 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:10.117874 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-m6zz7_6e4ab588-dddb-4af3-a430-a4722529b3b2/kube-rbac-proxy-main/0.log" Jul 23 18:44:10.155412 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:10.155382 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-m6zz7_6e4ab588-dddb-4af3-a430-a4722529b3b2/kube-rbac-proxy-self/0.log" Jul 23 18:44:10.216386 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:10.216358 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-pgnbs_e467ddca-d02c-4880-a006-9190caca249a/monitoring-plugin/0.log" Jul 23 18:44:10.255841 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:10.255800 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cvcp7_d487ee73-35f1-441c-8d5d-a7b54ec44244/node-exporter/0.log" Jul 23 18:44:10.279649 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:10.279624 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cvcp7_d487ee73-35f1-441c-8d5d-a7b54ec44244/kube-rbac-proxy/0.log" Jul 23 18:44:10.302904 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:10.302878 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cvcp7_d487ee73-35f1-441c-8d5d-a7b54ec44244/init-textfile/0.log" Jul 23 18:44:10.497743 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:10.497715 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-8djql_b81b721b-00d4-4745-95b3-e0f93171922e/kube-rbac-proxy-main/0.log" Jul 23 18:44:10.520171 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:10.520144 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-8djql_b81b721b-00d4-4745-95b3-e0f93171922e/kube-rbac-proxy-self/0.log" Jul 23 18:44:10.549080 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:10.549055 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-8djql_b81b721b-00d4-4745-95b3-e0f93171922e/openshift-state-metrics/0.log" Jul 23 18:44:10.956481 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:10.956397 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79644dc894-vbcpf_6a13fca3-602d-41b5-8057-fc0a786f13aa/thanos-query/0.log" Jul 23 18:44:10.979450 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:10.979425 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79644dc894-vbcpf_6a13fca3-602d-41b5-8057-fc0a786f13aa/kube-rbac-proxy-web/0.log" Jul 23 18:44:11.003903 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.003879 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79644dc894-vbcpf_6a13fca3-602d-41b5-8057-fc0a786f13aa/kube-rbac-proxy/0.log" Jul 23 18:44:11.028198 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.028174 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79644dc894-vbcpf_6a13fca3-602d-41b5-8057-fc0a786f13aa/prom-label-proxy/0.log" Jul 23 18:44:11.050697 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.050675 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79644dc894-vbcpf_6a13fca3-602d-41b5-8057-fc0a786f13aa/kube-rbac-proxy-rules/0.log" Jul 23 18:44:11.075630 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.075606 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79644dc894-vbcpf_6a13fca3-602d-41b5-8057-fc0a786f13aa/kube-rbac-proxy-metrics/0.log" Jul 23 18:44:11.767786 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.767746 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp"] Jul 23 18:44:11.768193 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.768094 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f255df26-62ae-4e34-8b0f-f743bcf583bf" containerName="test-curl-7585-140210904409120" Jul 23 18:44:11.768193 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.768107 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f255df26-62ae-4e34-8b0f-f743bcf583bf" containerName="test-curl-7585-140210904409120" Jul 23 18:44:11.768193 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.768178 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f255df26-62ae-4e34-8b0f-f743bcf583bf" containerName="test-curl-7585-140210904409120" Jul 23 18:44:11.771282 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.771261 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.774315 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.774301 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2mgnx\"/\"openshift-service-ca.crt\"" Jul 23 18:44:11.775555 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.775537 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2mgnx\"/\"default-dockercfg-vpwh6\"" Jul 23 18:44:11.775614 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.775587 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2mgnx\"/\"kube-root-ca.crt\"" Jul 23 18:44:11.780761 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.780741 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp"] Jul 23 18:44:11.824487 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.824461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwnts\" (UniqueName: \"kubernetes.io/projected/519c22d7-11f1-42eb-baed-ce7a61380e0d-kube-api-access-dwnts\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.824635 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.824496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/519c22d7-11f1-42eb-baed-ce7a61380e0d-lib-modules\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.824635 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.824523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/519c22d7-11f1-42eb-baed-ce7a61380e0d-podres\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.824635 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.824578 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/519c22d7-11f1-42eb-baed-ce7a61380e0d-proc\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.824635 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.824611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/519c22d7-11f1-42eb-baed-ce7a61380e0d-sys\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.925387 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.925357 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwnts\" (UniqueName: \"kubernetes.io/projected/519c22d7-11f1-42eb-baed-ce7a61380e0d-kube-api-access-dwnts\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.925387 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.925391 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/519c22d7-11f1-42eb-baed-ce7a61380e0d-lib-modules\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.925606 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.925412 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/519c22d7-11f1-42eb-baed-ce7a61380e0d-podres\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.925606 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.925436 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/519c22d7-11f1-42eb-baed-ce7a61380e0d-proc\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.925606 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.925476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/519c22d7-11f1-42eb-baed-ce7a61380e0d-sys\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.925606 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.925556 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/519c22d7-11f1-42eb-baed-ce7a61380e0d-podres\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.925606 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.925555 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/519c22d7-11f1-42eb-baed-ce7a61380e0d-lib-modules\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.925606 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.925591 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/519c22d7-11f1-42eb-baed-ce7a61380e0d-proc\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.925848 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.925602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/519c22d7-11f1-42eb-baed-ce7a61380e0d-sys\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:11.936580 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:11.936550 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwnts\" (UniqueName: \"kubernetes.io/projected/519c22d7-11f1-42eb-baed-ce7a61380e0d-kube-api-access-dwnts\") pod \"perf-node-gather-daemonset-7f7bp\" (UID: \"519c22d7-11f1-42eb-baed-ce7a61380e0d\") " pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:12.082368 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:12.082262 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:12.221502 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:12.221474 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp"] Jul 23 18:44:12.223189 ip-10-0-142-147 kubenswrapper[2564]: W0723 18:44:12.223161 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod519c22d7_11f1_42eb_baed_ce7a61380e0d.slice/crio-b205df9780dc16d6fafa02d3a8be747a4a2ddb797efc6d4a3fa05c99437b272b WatchSource:0}: Error finding container b205df9780dc16d6fafa02d3a8be747a4a2ddb797efc6d4a3fa05c99437b272b: Status 404 returned error can't find the container with id b205df9780dc16d6fafa02d3a8be747a4a2ddb797efc6d4a3fa05c99437b272b Jul 23 18:44:12.305138 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:12.305108 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-d997q_d178632a-a6f8-4d69-bf47-a4302711b7e9/networking-console-plugin/0.log" Jul 23 18:44:13.100318 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:13.100282 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" event={"ID":"519c22d7-11f1-42eb-baed-ce7a61380e0d","Type":"ContainerStarted","Data":"ca7786fd0d6ad6019ad3102d5fbe76ad3dce5183ea5d9a25114f9f0d2570d597"} Jul 23 18:44:13.100737 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:13.100344 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" event={"ID":"519c22d7-11f1-42eb-baed-ce7a61380e0d","Type":"ContainerStarted","Data":"b205df9780dc16d6fafa02d3a8be747a4a2ddb797efc6d4a3fa05c99437b272b"} Jul 23 18:44:13.100737 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:13.100373 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:13.122481 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:13.122435 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" podStartSLOduration=2.122418859 podStartE2EDuration="2.122418859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:44:13.120701301 +0000 UTC m=+2802.562199906" watchObservedRunningTime="2026-07-23 18:44:13.122418859 +0000 UTC m=+2802.563917455" Jul 23 18:44:14.086491 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:14.086463 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-n77rj_d49329fa-49a2-4376-82d6-aa2b7830f925/volume-data-source-validator/0.log" Jul 23 18:44:15.164466 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:15.164436 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5nhzx_ce29f6ec-8310-4439-96e9-adaec2684196/dns/0.log" Jul 23 18:44:15.188629 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:15.188588 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5nhzx_ce29f6ec-8310-4439-96e9-adaec2684196/kube-rbac-proxy/0.log" Jul 23 18:44:15.353383 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:15.353351 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gwvmh_d2dacd0d-9d63-4da0-ad08-1c36f4fa3dad/dns-node-resolver/0.log" Jul 23 18:44:15.972294 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:15.972265 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-db4467ff9-p7t8v_5431f723-ba76-49a7-ba80-df67a9d7e69b/registry/0.log" Jul 23 18:44:16.068721 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:16.068698 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x82j5_6098df05-0cf3-46ae-83ea-f16c1edf1260/node-ca/0.log" Jul 23 18:44:17.244568 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:17.244534 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cdff8959d-5s8vm_961f48be-e367-4efd-bbeb-570bf06a5d68/kube-auth-proxy/0.log" Jul 23 18:44:17.461711 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:17.461682 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-944979c7d-2pgzv_1fbd69ae-ab96-406e-97c8-d36b6c314b83/router/0.log" Jul 23 18:44:18.240459 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:18.240429 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hvkrn_dee23d54-ccac-4849-8abd-3f25a806ca2e/serve-healthcheck-canary/0.log" Jul 23 18:44:18.908357 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:18.908325 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-krgbq_e46a507c-b10b-4bac-8e85-01475e644526/kube-rbac-proxy/0.log" Jul 23 18:44:18.930781 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:18.930748 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-krgbq_e46a507c-b10b-4bac-8e85-01475e644526/exporter/0.log" Jul 23 18:44:18.952736 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:18.952707 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-krgbq_e46a507c-b10b-4bac-8e85-01475e644526/extractor/0.log" Jul 23 18:44:19.117573 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:19.117544 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2mgnx/perf-node-gather-daemonset-7f7bp" Jul 23 18:44:22.355691 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:22.355665 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f47b6774-kb9x5_79ee29b5-8549-413b-af02-2621002d17e2/manager/0.log" Jul 23 18:44:22.404389 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:22.404354 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f47b6774-kb9x5_79ee29b5-8549-413b-af02-2621002d17e2/copy-manifests/0.log" Jul 23 18:44:22.677916 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:22.677831 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cdff8959d-5s8vm_961f48be-e367-4efd-bbeb-570bf06a5d68/kube-auth-proxy/0.log" Jul 23 18:44:22.877674 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:22.877645 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-944979c7d-2pgzv_1fbd69ae-ab96-406e-97c8-d36b6c314b83/router/0.log" Jul 23 18:44:23.877220 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:23.877142 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-56f479bbd7-2mdsd_45bc571d-84a5-4af4-aa85-49b1b313371c/manager/0.log" Jul 23 18:44:24.279534 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:24.279500 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cdff8959d-5s8vm_961f48be-e367-4efd-bbeb-570bf06a5d68/kube-auth-proxy/0.log" Jul 23 18:44:24.448573 ip-10-0-142-147 kubenswrapper[2564]: I0723 18:44:24.448544 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-944979c7d-2pgzv_1fbd69ae-ab96-406e-97c8-d36b6c314b83/router/0.log"