Jul 08 22:46:45.630361 ip-10-0-143-210 systemd[1]: Starting Kubernetes Kubelet... Jul 08 22:46:46.100573 ip-10-0-143-210 systemd[1]: Started Kubernetes Kubelet. Jul 08 22:46:46.109226 ip-10-0-143-210 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 22:46:46.109226 ip-10-0-143-210 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 22:46:46.109226 ip-10-0-143-210 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 22:46:46.109226 ip-10-0-143-210 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 22:46:46.109226 ip-10-0-143-210 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 22:46:46.109226 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.035153 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 22:46:46.109226 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040146 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 22:46:46.109226 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040159 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 22:46:46.109226 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040162 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 22:46:46.109226 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040166 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 22:46:46.109226 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040169 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 22:46:46.109226 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040172 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 22:46:46.109226 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040175 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 22:46:46.109226 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040178 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040181 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040183 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040186 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040189 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040193 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040196 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040199 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040202 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040204 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040207 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040209 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040212 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040214 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040218 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040221 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040223 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040232 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040235 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040237 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 22:46:46.117537 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040240 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040243 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040247 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040251 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040254 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040256 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040259 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040261 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040264 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040267 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040269 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040272 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040274 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040279 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040283 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040286 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040289 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040291 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040294 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 22:46:46.118607 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040297 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040299 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040302 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040305 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040307 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040310 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040312 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040315 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040317 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040320 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040323 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040325 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040328 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040331 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040334 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040337 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040339 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040343 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040346 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040348 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 22:46:46.119677 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040352 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040354 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040357 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040359 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040362 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040365 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040369 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040372 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040375 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040378 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040381 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040384 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040388 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040391 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040394 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040396 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040399 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040402 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040404 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 22:46:46.120744 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040407 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040409 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040415 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040418 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040420 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040423 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040425 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040428 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040431 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040433 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040436 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040438 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040441 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.040444 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041181 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041189 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041194 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041198 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041201 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 22:46:46.121546 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041204 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041207 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041210 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041213 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041216 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041219 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041222 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041225 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041228 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041231 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041233 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041236 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041238 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041241 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041243 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041249 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041252 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041256 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041258 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041261 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 22:46:46.122423 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041264 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041266 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041269 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041272 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041274 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041277 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041279 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041282 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041285 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041287 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041290 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041293 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041296 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041298 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041301 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041304 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041307 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041310 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041314 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041317 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 22:46:46.123298 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041321 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041324 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041326 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041329 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041331 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041334 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041337 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041340 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041343 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041345 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041348 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041351 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041353 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041355 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041358 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041361 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041363 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041366 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041368 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041372 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 22:46:46.124362 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041374 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041377 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041380 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041382 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041385 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041387 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041390 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041392 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041395 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041397 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041403 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041406 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041410 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041412 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041415 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041418 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041420 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041423 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041426 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041432 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 22:46:46.125212 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041435 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041438 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041440 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041443 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041446 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041448 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041450 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041453 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041456 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041459 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041462 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041464 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041467 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041469 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041835 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041840 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041843 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041846 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041849 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 22:46:46.126074 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041851 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041854 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041857 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041859 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041862 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041865 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041867 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041870 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041873 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041875 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041878 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041880 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041884 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041887 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041889 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041892 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041895 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041897 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041900 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041903 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 22:46:46.126945 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041905 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041908 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041911 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041913 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041915 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041918 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041920 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041923 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041926 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041928 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041931 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041934 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041936 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041939 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041943 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041946 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041950 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041953 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041956 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 22:46:46.127818 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041958 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041961 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041964 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041967 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041969 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041975 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041977 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041980 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041984 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041988 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041992 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.041996 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042000 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042019 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042023 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042027 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042031 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042034 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042036 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042039 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 22:46:46.128693 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042042 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042045 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042048 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042051 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042053 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042056 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042059 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042062 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042065 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042068 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042071 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042074 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042076 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042092 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042095 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042098 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042101 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042107 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042110 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 22:46:46.129750 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042112 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042115 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042117 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042121 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042124 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042126 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042129 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042132 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042134 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042137 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042140 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042142 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042145 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042147 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042150 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042152 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042226 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042233 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042240 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042244 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042248 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 22:46:46.130625 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042251 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042262 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042266 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042270 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042273 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042277 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042280 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042283 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042285 2564 flags.go:64] FLAG: --cgroup-root="" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042291 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042294 2564 flags.go:64] FLAG: --client-ca-file="" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042297 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042300 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042304 2564 flags.go:64] FLAG: --cluster-domain="" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042307 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042310 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042313 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042316 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042320 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042323 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042326 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042329 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042332 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042335 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 22:46:46.131754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042338 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042341 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042344 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042348 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042351 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042354 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042357 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042360 2564 flags.go:64] FLAG: --enable-server="true" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042362 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042367 2564 flags.go:64] FLAG: --event-burst="100" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042370 2564 flags.go:64] FLAG: --event-qps="50" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042374 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042378 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042381 2564 flags.go:64] FLAG: --eviction-hard="" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042384 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042387 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042390 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042396 2564 flags.go:64] FLAG: --eviction-soft="" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042399 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042402 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042405 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042408 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042411 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042414 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042416 2564 flags.go:64] FLAG: --feature-gates="" Jul 08 22:46:46.132844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042420 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042422 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042426 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042429 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042432 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042435 2564 flags.go:64] FLAG: --help="false" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042437 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042441 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042443 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042446 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042449 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042452 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042455 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042458 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042460 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042463 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042466 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042469 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042472 2564 flags.go:64] FLAG: --kube-reserved="" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042474 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042478 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042481 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042483 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042486 2564 flags.go:64] FLAG: --lock-file="" Jul 08 22:46:46.134581 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042491 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042494 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042497 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042502 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042505 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042508 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042511 2564 flags.go:64] FLAG: --logging-format="text" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042513 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042516 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042520 2564 flags.go:64] FLAG: --manifest-url="" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042522 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042527 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042530 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042533 2564 flags.go:64] FLAG: --max-pods="110" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042536 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042539 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042542 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042545 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042548 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042550 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042553 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042561 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042564 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042567 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 22:46:46.135673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042570 2564 flags.go:64] FLAG: --pod-cidr="" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042573 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042578 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042581 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042584 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042587 2564 flags.go:64] FLAG: --port="10250" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042590 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042593 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-036cdb0770f649cd5" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042597 2564 flags.go:64] FLAG: --qos-reserved="" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042601 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042604 2564 flags.go:64] FLAG: --register-node="true" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042607 2564 flags.go:64] FLAG: --register-with-taints="" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042610 2564 flags.go:64] FLAG: --registry-burst="10" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042613 2564 flags.go:64] FLAG: --registry-qps="5" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042616 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042619 2564 flags.go:64] FLAG: --reserved-memory="" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042622 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042625 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042628 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042631 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042634 2564 flags.go:64] FLAG: --runonce="false" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042637 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042640 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042643 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042645 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 22:46:46.136749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042648 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042651 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042654 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042657 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042659 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042662 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042665 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042667 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042670 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042673 2564 flags.go:64] FLAG: --system-cgroups="" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042676 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042681 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042683 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042687 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042691 2564 flags.go:64] FLAG: --tls-min-version="" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042694 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042698 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042701 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042704 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042707 2564 flags.go:64] FLAG: --v="2" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042710 2564 flags.go:64] FLAG: --version="false" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042714 2564 flags.go:64] FLAG: --vmodule="" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042718 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.042721 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042874 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 22:46:46.275826 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042879 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042882 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042885 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042888 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042891 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042894 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042896 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042899 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042902 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042904 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042907 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042910 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042912 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042915 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042918 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042920 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042923 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042925 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042928 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042931 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 22:46:46.294669 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042933 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042936 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042940 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042945 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042947 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042950 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042952 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042955 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042957 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042960 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042962 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042965 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042967 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042970 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042973 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042975 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042978 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042980 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042983 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042985 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 22:46:46.302077 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042988 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042991 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042993 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042995 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.042998 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043000 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043003 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043005 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043008 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043011 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043015 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043018 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043021 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043023 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043028 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043032 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043035 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043037 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043040 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043042 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 22:46:46.303342 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043045 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043049 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043052 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043055 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043058 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043061 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043063 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043066 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043069 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043071 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043075 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043091 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043094 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043098 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043100 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043103 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043106 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043108 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043110 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 22:46:46.304391 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043113 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043115 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043118 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043120 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043123 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043125 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043128 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043132 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043136 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043139 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043142 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043145 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043147 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043150 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043153 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043155 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043158 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043160 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 22:46:46.305477 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.043163 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 22:46:46.306369 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.044017 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 22:46:46.306369 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.050869 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 22:46:46.306369 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.050885 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 22:46:46.306369 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.050897 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 22:46:46.306369 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.050907 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 22:46:46.306369 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.050962 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 22:46:46.306369 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.050967 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 22:46:46.306369 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.050971 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 22:46:46.306369 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.050974 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 22:46:46.306369 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.050977 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 22:46:46.306369 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.050979 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 22:46:46.306369 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.050983 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 22:46:46.306369 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.050986 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 22:46:46.306369 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.050989 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.050991 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.050994 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.050996 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.050999 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051001 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051004 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051007 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051010 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051013 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051015 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051018 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051021 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051023 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051026 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051029 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051032 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051034 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051037 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051039 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 22:46:46.307461 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051042 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051045 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051048 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051051 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051054 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051057 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051060 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051063 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051065 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051068 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051071 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051073 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051076 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051093 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051096 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051101 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051105 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051108 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051111 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 22:46:46.308338 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051114 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051121 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051130 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051132 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051135 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051138 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051140 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051144 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051148 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051151 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051154 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051157 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051160 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051163 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051166 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051169 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051171 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051174 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051177 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051181 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 22:46:46.309440 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051185 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051188 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051191 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051193 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051196 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051198 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051204 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051207 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051209 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051212 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051215 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051217 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051220 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051222 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051225 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051228 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051231 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051233 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051236 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051238 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 22:46:46.310315 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051241 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051243 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051246 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051248 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051251 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051255 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051257 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051260 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051262 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051265 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051268 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051270 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.051276 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051387 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051391 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 22:46:46.311211 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051394 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051396 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051399 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051402 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051407 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051410 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051412 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051416 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051420 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051423 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051426 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051429 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051431 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051434 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051437 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051440 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051442 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051445 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051447 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051450 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 22:46:46.311883 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051452 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051455 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051460 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051463 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051467 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051470 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051473 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051475 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051478 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051480 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051483 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051485 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051488 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051491 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051493 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051496 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051500 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051502 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051505 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 22:46:46.312984 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051507 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051510 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051512 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051516 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051519 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051522 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051525 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051528 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051530 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051533 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051535 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051538 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051541 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051543 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051546 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051550 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051552 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051555 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051557 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051560 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 22:46:46.313888 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051562 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051565 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051568 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051570 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051573 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051576 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051578 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051580 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051583 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051587 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051589 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051592 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051595 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051597 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051599 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051602 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051604 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051607 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051610 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 22:46:46.314765 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051612 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051615 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051617 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051620 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051622 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051625 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051627 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051629 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051633 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051636 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051638 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051641 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051644 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051646 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051649 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051651 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051654 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051656 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 22:46:46.315639 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:46.051659 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 22:46:46.316723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.051664 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 22:46:46.316723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.051768 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 22:46:46.316723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.053862 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 22:46:46.316723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.055107 2564 server.go:1019] "Starting client certificate rotation" Jul 08 22:46:46.316723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.055222 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 22:46:46.316723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.055320 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 22:46:46.316723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.058443 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 22:46:46.316723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.061584 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 22:46:46.316723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.068098 2564 log.go:25] "Validated CRI v1 runtime API" Jul 08 22:46:46.316723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.069952 2564 log.go:25] "Validated CRI v1 image API" Jul 08 22:46:46.316723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.070718 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 22:46:46.316723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.072859 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c159a17b-b03f-4b58-b11b-8341c228dc5a:/dev/nvme0n1p4 d9ed1dbe-bf5b-469e-abcb-e08ca9d8e12c:/dev/nvme0n1p3] Jul 08 22:46:46.317409 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.072881 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 22:46:46.317648 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.076958 2564 manager.go:216] Machine: {Timestamp:2026-07-08 22:46:46.076495565 +0000 UTC m=+0.331943259 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099008 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a6d5606bd7d0050c1957c48532dd7 SystemUUID:ec2a6d56-06bd-7d00-50c1-957c48532dd7 BootID:c873339b-a9e3-48d2-aa41-9cbcd8c8ec8b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:54:78:f6:49:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:54:78:f6:49:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:d5:d1:23:79:74 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 22:46:46.317648 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.077054 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 22:46:46.317648 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.077157 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 22:46:46.317648 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.079483 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 22:46:46.317648 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.079509 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-210.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 22:46:46.317648 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.080141 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 22:46:46.317648 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.080151 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 22:46:46.317648 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.080159 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.081399 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.082664 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.085857 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.086045 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.086613 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.086627 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.086642 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.086654 2564 kubelet.go:398] "Adding apiserver pod source" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.086668 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.087055 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.087075 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.089902 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.091647 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.091674 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.093152 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.093169 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.093175 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.093181 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.093190 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.093195 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.093204 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.093209 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.093217 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.093222 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.093230 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.093235 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.093255 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.093260 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.094947 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.096285 2564 server.go:1268] "Started kubelet" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.096341 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.096355 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.096422 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.100299 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.102959 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-210.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.103557 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.104396 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 22:46:46.318124 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.104403 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.106781 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-210.ec2.internal.18c072022c3aadd8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-210.ec2.internal,UID:ip-10-0-143-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-210.ec2.internal,},FirstTimestamp:2026-07-08 22:46:46.096260568 +0000 UTC m=+0.351708262,LastTimestamp:2026-07-08 22:46:46.096260568 +0000 UTC m=+0.351708262,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-210.ec2.internal,}" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.111846 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.111850 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.111980 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.112697 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.112713 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.112696 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.112862 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.112872 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.113428 2564 factory.go:153] Registering CRI-O factory Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.113444 2564 factory.go:223] Registration of the crio container factory successfully Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.113497 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.113507 2564 factory.go:55] Registering systemd factory Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.113516 2564 factory.go:223] Registration of the systemd container factory successfully Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.113537 2564 factory.go:103] Registering Raw factory Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.113551 2564 manager.go:1196] Started watching for new ooms in manager Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.114816 2564 manager.go:349] Starting recovery of all containers Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.115897 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.115962 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.124689 2564 manager.go:354] Recovery completed Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.130133 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.132010 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.132036 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.132045 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.132726 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.132692 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.133163 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.133179 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.133440 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.133465 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 22:46:46.319891 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.133475 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.133938 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-210.ec2.internal.18c072022e5c71a1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-210.ec2.internal,UID:ip-10-0-143-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-210.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-210.ec2.internal,},FirstTimestamp:2026-07-08 22:46:46.132027809 +0000 UTC m=+0.387475502,LastTimestamp:2026-07-08 22:46:46.132027809 +0000 UTC m=+0.387475502,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-210.ec2.internal,}" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.142443 2564 policy_none.go:49] "None policy: Start" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.142456 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.142467 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.143708 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fpq95" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.143725 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fpq95" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.144198 2564 policy_none.go:47] "Start" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.144984 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-210.ec2.internal.18c072022e5ca2b6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-210.ec2.internal,UID:ip-10-0-143-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-210.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-210.ec2.internal,},FirstTimestamp:2026-07-08 22:46:46.132040374 +0000 UTC m=+0.387488069,LastTimestamp:2026-07-08 22:46:46.132040374 +0000 UTC m=+0.387488069,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-210.ec2.internal,}" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.188801 2564 manager.go:337] "Starting Device Plugin manager" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.188944 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.188959 2564 server.go:87] "Starting device plugin registration server" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.189177 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.189187 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.189263 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.189344 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.189353 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.189747 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.189800 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.259820 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.261072 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.261117 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.261140 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.261173 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.263290 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.290110 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.292010 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 08 22:46:46.323566 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.292040 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.292055 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.292095 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.301028 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.301046 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-210.ec2.internal\": node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.322003 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.361748 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal"] Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.361821 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.363179 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.363204 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.363214 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.365337 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.365460 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.365493 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.366029 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.366057 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.366066 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.366029 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.366123 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.366135 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.368254 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.368275 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.368943 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.368963 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.368974 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.403703 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-210.ec2.internal\" not found" node="ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.408108 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-210.ec2.internal\" not found" node="ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.414294 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1f8a59a8baac1caed8261311e43cf1fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"1f8a59a8baac1caed8261311e43cf1fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.461064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.414314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f8a59a8baac1caed8261311e43cf1fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"1f8a59a8baac1caed8261311e43cf1fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.462736 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.414329 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be9286b9c4dca1e8d98a40958054271d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-210.ec2.internal\" (UID: \"be9286b9c4dca1e8d98a40958054271d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.462736 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.422333 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:46.639672 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.515155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be9286b9c4dca1e8d98a40958054271d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-210.ec2.internal\" (UID: \"be9286b9c4dca1e8d98a40958054271d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.639672 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.515195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1f8a59a8baac1caed8261311e43cf1fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"1f8a59a8baac1caed8261311e43cf1fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.639672 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.515215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f8a59a8baac1caed8261311e43cf1fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"1f8a59a8baac1caed8261311e43cf1fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.639672 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.515262 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f8a59a8baac1caed8261311e43cf1fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"1f8a59a8baac1caed8261311e43cf1fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.639672 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.515278 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be9286b9c4dca1e8d98a40958054271d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-210.ec2.internal\" (UID: \"be9286b9c4dca1e8d98a40958054271d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.639672 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.515284 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1f8a59a8baac1caed8261311e43cf1fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"1f8a59a8baac1caed8261311e43cf1fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.639672 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.522509 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:46.639672 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.622855 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:46.765925 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.709541 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.765925 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:46.711867 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" Jul 08 22:46:46.765925 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.723577 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:46.906936 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.823908 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:47.016171 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:46.924128 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:47.028497 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:47.024890 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:47.157867 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.056385 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 22:46:47.157867 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.056524 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 22:46:47.157867 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.119963 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 22:46:47.157867 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:47.125162 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:47.157867 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.146560 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 22:41:46 +0000 UTC" deadline="2028-02-20 11:34:28.029412372 +0000 UTC" Jul 08 22:46:47.157867 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.146589 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14196h47m40.882826693s" Jul 08 22:46:47.174313 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.170486 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 22:46:47.174313 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.171955 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 08 22:46:47.174313 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.171983 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 22:46:47.174313 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.171993 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 08 22:46:47.303738 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:47.226208 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:47.429182 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:47.326992 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:47.429182 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:47.427691 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:47.569075 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:47.528420 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 08 22:46:47.694269 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.628210 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 22:46:47.694269 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.656628 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 22:46:47.812711 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.712769 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 08 22:46:47.812711 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.725420 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 22:46:47.812711 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.726464 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" Jul 08 22:46:47.812711 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:47.732702 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 22:46:48.069414 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:48.069371 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9286b9c4dca1e8d98a40958054271d.slice/crio-0c8c3fadd4024452399b87630229e36e1a7574d6087987b90cc6a7763b07e705 WatchSource:0}: Error finding container 0c8c3fadd4024452399b87630229e36e1a7574d6087987b90cc6a7763b07e705: Status 404 returned error can't find the container with id 0c8c3fadd4024452399b87630229e36e1a7574d6087987b90cc6a7763b07e705 Jul 08 22:46:48.075255 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.075234 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 22:46:48.081801 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:48.081773 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8a59a8baac1caed8261311e43cf1fc.slice/crio-0928363e829b5154a82730cda6a3dea195c8550b4fa0990c61d748b4c82525fa WatchSource:0}: Error finding container 0928363e829b5154a82730cda6a3dea195c8550b4fa0990c61d748b4c82525fa: Status 404 returned error can't find the container with id 0928363e829b5154a82730cda6a3dea195c8550b4fa0990c61d748b4c82525fa Jul 08 22:46:48.091474 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.091459 2564 apiserver.go:52] "Watching apiserver" Jul 08 22:46:48.099333 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.099313 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 22:46:48.101019 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.100998 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-grqlf","kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5","openshift-cluster-node-tuning-operator/tuned-9t6cr","openshift-dns/node-resolver-297fk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal","openshift-multus/multus-j6kcc","openshift-multus/network-metrics-daemon-rldqz","kube-system/global-pull-secret-syncer-twk7h","openshift-image-registry/node-ca-h7dmt","openshift-multus/multus-additional-cni-plugins-gjw2b","openshift-network-diagnostics/network-check-target-jh8dz","openshift-network-operator/iptables-alerter-qjtkc","openshift-ovn-kubernetes/ovnkube-node-pd2mh"] Jul 08 22:46:48.103857 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.103838 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.106027 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.106006 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qjtkc" Jul 08 22:46:48.106531 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.106510 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4l6bv\"" Jul 08 22:46:48.106620 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.106575 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 22:46:48.106620 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.106586 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 22:46:48.106727 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.106681 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 22:46:48.106788 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.106721 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 22:46:48.108337 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.108317 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:48.108436 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.108411 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:46:48.108713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.108687 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 22:46:48.108815 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.108745 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gf7ql\"" Jul 08 22:46:48.108815 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.108757 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 22:46:48.108965 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.108902 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 22:46:48.110719 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.110690 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.112564 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.112549 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 22:46:48.112770 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.112755 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q4rfx\"" Jul 08 22:46:48.113371 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.112978 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 22:46:48.113371 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.113042 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-297fk" Jul 08 22:46:48.113371 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.113050 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 22:46:48.115421 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.115402 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:46:48.115513 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.115458 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:46:48.116631 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.116580 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 22:46:48.116732 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.116634 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nffww\"" Jul 08 22:46:48.117171 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.117142 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 22:46:48.119183 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.119156 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.121380 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.121355 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-69dvr\"" Jul 08 22:46:48.121685 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.121671 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h7dmt" Jul 08 22:46:48.122257 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.122103 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 22:46:48.122257 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.122109 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 22:46:48.123136 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123074 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46b588ae-74ca-487b-bc6b-8db2fb29691d-tmp\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.123240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-cnibin\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.123240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-os-release\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.123362 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123248 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-run-netns\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.123362 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-var-lib-cni-bin\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.123362 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-multus-conf-dir\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.123362 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-run-multus-certs\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.123362 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/150e8874-8d0a-4fb3-8079-bb87507a4393-host-slash\") pod \"iptables-alerter-qjtkc\" (UID: \"150e8874-8d0a-4fb3-8079-bb87507a4393\") " pod="openshift-network-operator/iptables-alerter-qjtkc" Jul 08 22:46:48.123592 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123393 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-systemd\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.123592 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaf82025-12e7-4d8d-a76a-127dd040d050-tmp-dir\") pod \"node-resolver-297fk\" (UID: \"eaf82025-12e7-4d8d-a76a-127dd040d050\") " pod="openshift-dns/node-resolver-297fk" Jul 08 22:46:48.123592 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123444 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrf6\" (UniqueName: \"kubernetes.io/projected/4ed6eae7-7b43-47ba-b489-136fab2a8ade-kube-api-access-mjrf6\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:48.123592 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123467 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-run\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.123592 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123490 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-hostroot\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.123592 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-modprobe-d\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.123592 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-lib-modules\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.123592 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123579 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-tuned\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.123917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-system-cni-dir\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.123917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123613 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c226b53-16d4-436d-8cae-e37ab29dafeb-cni-binary-copy\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.123917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123629 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/150e8874-8d0a-4fb3-8079-bb87507a4393-iptables-alerter-script\") pod \"iptables-alerter-qjtkc\" (UID: \"150e8874-8d0a-4fb3-8079-bb87507a4393\") " pod="openshift-network-operator/iptables-alerter-qjtkc" Jul 08 22:46:48.123917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-sysconfig\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.123917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-sysctl-conf\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.123917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123724 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-var-lib-kubelet\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.123917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123764 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-multus-cni-dir\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.123917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123796 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-var-lib-cni-multus\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.123917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123842 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c226b53-16d4-436d-8cae-e37ab29dafeb-multus-daemon-config\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.123917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123862 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 22:46:48.123917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gmds\" (UniqueName: \"kubernetes.io/projected/5c226b53-16d4-436d-8cae-e37ab29dafeb-kube-api-access-4gmds\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123930 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-kubernetes\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123967 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz59w\" (UniqueName: \"kubernetes.io/projected/eaf82025-12e7-4d8d-a76a-127dd040d050-kube-api-access-kz59w\") pod \"node-resolver-297fk\" (UID: \"eaf82025-12e7-4d8d-a76a-127dd040d050\") " pod="openshift-dns/node-resolver-297fk" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.123998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.124024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-sys\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.124045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-host\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.124070 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz8bp\" (UniqueName: \"kubernetes.io/projected/46b588ae-74ca-487b-bc6b-8db2fb29691d-kube-api-access-vz8bp\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.124075 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.124096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eaf82025-12e7-4d8d-a76a-127dd040d050-hosts-file\") pod \"node-resolver-297fk\" (UID: \"eaf82025-12e7-4d8d-a76a-127dd040d050\") " pod="openshift-dns/node-resolver-297fk" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.124140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-multus-socket-dir-parent\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.124191 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-var-lib-kubelet\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.124216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-etc-kubernetes\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.124240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-sysctl-d\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.124264 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-run-k8s-cni-cncf-io\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.124302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88gkf\" (UniqueName: \"kubernetes.io/projected/150e8874-8d0a-4fb3-8079-bb87507a4393-kube-api-access-88gkf\") pod \"iptables-alerter-qjtkc\" (UID: \"150e8874-8d0a-4fb3-8079-bb87507a4393\") " pod="openshift-network-operator/iptables-alerter-qjtkc" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.124341 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-grqlf" Jul 08 22:46:48.124443 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.124416 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 22:46:48.124989 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.124352 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h29sg\"" Jul 08 22:46:48.126460 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.126437 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 22:46:48.126592 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.126574 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:48.126592 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.126584 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qltsr\"" Jul 08 22:46:48.126699 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.126636 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:46:48.126884 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.126866 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 22:46:48.127363 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.127343 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 22:46:48.129053 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.129040 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.131255 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.131232 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 22:46:48.131468 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.131445 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 22:46:48.131579 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.131468 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bqfgf\"" Jul 08 22:46:48.131579 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.131510 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 22:46:48.131689 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.131665 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.134777 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.134757 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 22:46:48.134885 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.134790 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rk6r5\"" Jul 08 22:46:48.134885 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.134796 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 22:46:48.134885 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.134802 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 22:46:48.134885 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.134828 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 22:46:48.135129 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.135105 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 22:46:48.135238 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.135137 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 22:46:48.149716 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.149697 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-47qg6" Jul 08 22:46:48.159810 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.159790 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-47qg6" Jul 08 22:46:48.214570 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.214503 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 22:46:48.225520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-var-lib-kubelet\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.225520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225519 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-multus-cni-dir\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.225714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-var-lib-cni-multus\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.225714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gmds\" (UniqueName: \"kubernetes.io/projected/5c226b53-16d4-436d-8cae-e37ab29dafeb-kube-api-access-4gmds\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.225714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-socket-dir\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.225714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225598 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-var-lib-openvswitch\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.225714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225612 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-var-lib-cni-multus\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.225714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63c5c2ac-8599-49be-b250-926ec607c77e-ovn-node-metrics-cert\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.225714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225660 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-var-lib-kubelet\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.225714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmd9f\" (UniqueName: \"kubernetes.io/projected/9c0a262a-d0f9-4fd3-91dc-44848d6d50b2-kube-api-access-dmd9f\") pod \"node-ca-h7dmt\" (UID: \"9c0a262a-d0f9-4fd3-91dc-44848d6d50b2\") " pod="openshift-image-registry/node-ca-h7dmt" Jul 08 22:46:48.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-kubernetes\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225749 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-multus-cni-dir\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz59w\" (UniqueName: \"kubernetes.io/projected/eaf82025-12e7-4d8d-a76a-127dd040d050-kube-api-access-kz59w\") pod \"node-resolver-297fk\" (UID: \"eaf82025-12e7-4d8d-a76a-127dd040d050\") " pod="openshift-dns/node-resolver-297fk" Jul 08 22:46:48.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:48.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225807 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-kubernetes\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-device-dir\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225865 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-log-socket\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-cni-bin\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-sys\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-multus-socket-dir-parent\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225950 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-var-lib-kubelet\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225975 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:48.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.225992 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqn7h\" (UniqueName: \"kubernetes.io/projected/df762a3b-8602-4846-be8c-32ac8fd3b237-kube-api-access-hqn7h\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226010 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-run-ovn\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.226783 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226009 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-multus-socket-dir-parent\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.226783 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226030 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-sys\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.226783 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226034 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz8gn\" (UniqueName: \"kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn\") pod \"network-check-target-jh8dz\" (UID: \"2a5fca6f-9523-47c5-b7f4-c6f45277ce47\") " pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:46:48.226783 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.226008 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 22:46:48.226783 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226054 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-var-lib-kubelet\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.226783 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-run-k8s-cni-cncf-io\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.226783 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.226134 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs podName:4ed6eae7-7b43-47ba-b489-136fab2a8ade nodeName:}" failed. No retries permitted until 2026-07-08 22:46:48.726086685 +0000 UTC m=+2.981534371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs") pod "network-metrics-daemon-rldqz" (UID: "4ed6eae7-7b43-47ba-b489-136fab2a8ade") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 22:46:48.226783 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226144 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-run-k8s-cni-cncf-io\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.226783 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-sys-fs\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.226783 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.226783 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226236 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63c5c2ac-8599-49be-b250-926ec607c77e-env-overrides\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.226783 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c0a262a-d0f9-4fd3-91dc-44848d6d50b2-host\") pod \"node-ca-h7dmt\" (UID: \"9c0a262a-d0f9-4fd3-91dc-44848d6d50b2\") " pod="openshift-image-registry/node-ca-h7dmt" Jul 08 22:46:48.226783 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226278 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46b588ae-74ca-487b-bc6b-8db2fb29691d-tmp\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.226783 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-cnibin\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-os-release\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-multus-conf-dir\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226353 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrf6\" (UniqueName: \"kubernetes.io/projected/4ed6eae7-7b43-47ba-b489-136fab2a8ade-kube-api-access-mjrf6\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226365 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-cnibin\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-etc-selinux\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226397 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-multus-conf-dir\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226425 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-os-release\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226491 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-node-log\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaf82025-12e7-4d8d-a76a-127dd040d050-tmp-dir\") pod \"node-resolver-297fk\" (UID: \"eaf82025-12e7-4d8d-a76a-127dd040d050\") " pod="openshift-dns/node-resolver-297fk" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e72ff1d4-bb03-44e5-80cb-8f8002a894e8-konnectivity-ca\") pod \"konnectivity-agent-grqlf\" (UID: \"e72ff1d4-bb03-44e5-80cb-8f8002a894e8\") " pod="kube-system/konnectivity-agent-grqlf" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-hostroot\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226611 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226616 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-hostroot\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226630 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e72ff1d4-bb03-44e5-80cb-8f8002a894e8-agent-certs\") pod \"konnectivity-agent-grqlf\" (UID: \"e72ff1d4-bb03-44e5-80cb-8f8002a894e8\") " pod="kube-system/konnectivity-agent-grqlf" Jul 08 22:46:48.227520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0677a18f-b178-455f-a982-4c31ec52aab0-dbus\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:48.228271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-registration-dir\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.228271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-cni-netd\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.228271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4m68\" (UniqueName: \"kubernetes.io/projected/63c5c2ac-8599-49be-b250-926ec607c77e-kube-api-access-q4m68\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.228271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3168d934-7d87-498f-9733-8059ff036149-system-cni-dir\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.228271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/150e8874-8d0a-4fb3-8079-bb87507a4393-iptables-alerter-script\") pod \"iptables-alerter-qjtkc\" (UID: \"150e8874-8d0a-4fb3-8079-bb87507a4393\") " pod="openshift-network-operator/iptables-alerter-qjtkc" Jul 08 22:46:48.228271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226879 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0677a18f-b178-455f-a982-4c31ec52aab0-kubelet-config\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:48.228271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226890 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaf82025-12e7-4d8d-a76a-127dd040d050-tmp-dir\") pod \"node-resolver-297fk\" (UID: \"eaf82025-12e7-4d8d-a76a-127dd040d050\") " pod="openshift-dns/node-resolver-297fk" Jul 08 22:46:48.228271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226895 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-run-systemd\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.228271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.226992 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-etc-openvswitch\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.228271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227029 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-sysconfig\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.228271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-sysctl-conf\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.228271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c226b53-16d4-436d-8cae-e37ab29dafeb-multus-daemon-config\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.228271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227079 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-run-ovn-kubernetes\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.228271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227136 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3168d934-7d87-498f-9733-8059ff036149-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c0a262a-d0f9-4fd3-91dc-44848d6d50b2-serviceca\") pod \"node-ca-h7dmt\" (UID: \"9c0a262a-d0f9-4fd3-91dc-44848d6d50b2\") " pod="openshift-image-registry/node-ca-h7dmt" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227212 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-sysctl-conf\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227220 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-sysconfig\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-kubelet\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227272 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-host\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8bp\" (UniqueName: \"kubernetes.io/projected/46b588ae-74ca-487b-bc6b-8db2fb29691d-kube-api-access-vz8bp\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eaf82025-12e7-4d8d-a76a-127dd040d050-hosts-file\") pod \"node-resolver-297fk\" (UID: \"eaf82025-12e7-4d8d-a76a-127dd040d050\") " pod="openshift-dns/node-resolver-297fk" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227330 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-host\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-etc-kubernetes\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227374 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eaf82025-12e7-4d8d-a76a-127dd040d050-hosts-file\") pod \"node-resolver-297fk\" (UID: \"eaf82025-12e7-4d8d-a76a-127dd040d050\") " pod="openshift-dns/node-resolver-297fk" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88gkf\" (UniqueName: \"kubernetes.io/projected/150e8874-8d0a-4fb3-8079-bb87507a4393-kube-api-access-88gkf\") pod \"iptables-alerter-qjtkc\" (UID: \"150e8874-8d0a-4fb3-8079-bb87507a4393\") " pod="openshift-network-operator/iptables-alerter-qjtkc" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-slash\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227387 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-etc-kubernetes\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-run-openvswitch\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.228713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-sysctl-d\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.229152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.229152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227510 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/150e8874-8d0a-4fb3-8079-bb87507a4393-iptables-alerter-script\") pod \"iptables-alerter-qjtkc\" (UID: \"150e8874-8d0a-4fb3-8079-bb87507a4393\") " pod="openshift-network-operator/iptables-alerter-qjtkc" Jul 08 22:46:48.229152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227534 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63c5c2ac-8599-49be-b250-926ec607c77e-ovnkube-script-lib\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.229152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3168d934-7d87-498f-9733-8059ff036149-cni-binary-copy\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.229152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227595 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-sysctl-d\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.229152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227613 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmh7\" (UniqueName: \"kubernetes.io/projected/3168d934-7d87-498f-9733-8059ff036149-kube-api-access-ncmh7\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.229152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-run-netns\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.229152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227654 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c226b53-16d4-436d-8cae-e37ab29dafeb-multus-daemon-config\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.229152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-var-lib-cni-bin\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.229152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227695 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-run-netns\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.229152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-run-multus-certs\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.229152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227741 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/150e8874-8d0a-4fb3-8079-bb87507a4393-host-slash\") pod \"iptables-alerter-qjtkc\" (UID: \"150e8874-8d0a-4fb3-8079-bb87507a4393\") " pod="openshift-network-operator/iptables-alerter-qjtkc" Jul 08 22:46:48.229152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227749 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-var-lib-cni-bin\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.229152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227756 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-host-run-multus-certs\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.229596 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-run-netns\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.229596 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227783 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/150e8874-8d0a-4fb3-8079-bb87507a4393-host-slash\") pod \"iptables-alerter-qjtkc\" (UID: \"150e8874-8d0a-4fb3-8079-bb87507a4393\") " pod="openshift-network-operator/iptables-alerter-qjtkc" Jul 08 22:46:48.229596 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227798 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63c5c2ac-8599-49be-b250-926ec607c77e-ovnkube-config\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.229596 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227828 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3168d934-7d87-498f-9733-8059ff036149-os-release\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.229596 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227857 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-systemd\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.229596 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3168d934-7d87-498f-9733-8059ff036149-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.229596 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227916 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-run\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.229596 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-systemd-units\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.229596 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227957 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-systemd\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.229596 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.227970 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3168d934-7d87-498f-9733-8059ff036149-cnibin\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.229596 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.228000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-modprobe-d\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.229596 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.228025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-lib-modules\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.229596 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.228028 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-run\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.229596 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.228044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-tuned\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.230019 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.228066 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-system-cni-dir\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.230019 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.228101 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-modprobe-d\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.230019 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.228107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c226b53-16d4-436d-8cae-e37ab29dafeb-cni-binary-copy\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.230019 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.228132 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46b588ae-74ca-487b-bc6b-8db2fb29691d-lib-modules\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.230019 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.228137 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c226b53-16d4-436d-8cae-e37ab29dafeb-system-cni-dir\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.230019 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.228150 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3168d934-7d87-498f-9733-8059ff036149-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.230019 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.228482 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c226b53-16d4-436d-8cae-e37ab29dafeb-cni-binary-copy\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.230019 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.229528 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46b588ae-74ca-487b-bc6b-8db2fb29691d-tmp\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.230019 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.229964 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46b588ae-74ca-487b-bc6b-8db2fb29691d-etc-tuned\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.234323 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.234303 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz59w\" (UniqueName: \"kubernetes.io/projected/eaf82025-12e7-4d8d-a76a-127dd040d050-kube-api-access-kz59w\") pod \"node-resolver-297fk\" (UID: \"eaf82025-12e7-4d8d-a76a-127dd040d050\") " pod="openshift-dns/node-resolver-297fk" Jul 08 22:46:48.234419 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.234306 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gmds\" (UniqueName: \"kubernetes.io/projected/5c226b53-16d4-436d-8cae-e37ab29dafeb-kube-api-access-4gmds\") pod \"multus-j6kcc\" (UID: \"5c226b53-16d4-436d-8cae-e37ab29dafeb\") " pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.234497 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.234480 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrf6\" (UniqueName: \"kubernetes.io/projected/4ed6eae7-7b43-47ba-b489-136fab2a8ade-kube-api-access-mjrf6\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:48.238232 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.238204 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz8bp\" (UniqueName: \"kubernetes.io/projected/46b588ae-74ca-487b-bc6b-8db2fb29691d-kube-api-access-vz8bp\") pod \"tuned-9t6cr\" (UID: \"46b588ae-74ca-487b-bc6b-8db2fb29691d\") " pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.238532 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.238516 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88gkf\" (UniqueName: \"kubernetes.io/projected/150e8874-8d0a-4fb3-8079-bb87507a4393-kube-api-access-88gkf\") pod \"iptables-alerter-qjtkc\" (UID: \"150e8874-8d0a-4fb3-8079-bb87507a4393\") " pod="openshift-network-operator/iptables-alerter-qjtkc" Jul 08 22:46:48.265879 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.265839 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" event={"ID":"1f8a59a8baac1caed8261311e43cf1fc","Type":"ContainerStarted","Data":"0928363e829b5154a82730cda6a3dea195c8550b4fa0990c61d748b4c82525fa"} Jul 08 22:46:48.266942 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.266918 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" event={"ID":"be9286b9c4dca1e8d98a40958054271d","Type":"ContainerStarted","Data":"0c8c3fadd4024452399b87630229e36e1a7574d6087987b90cc6a7763b07e705"} Jul 08 22:46:48.328785 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.328751 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-device-dir\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.328785 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.328786 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-log-socket\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.329025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.328809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-cni-bin\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.329025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.328834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:48.329025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.328857 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqn7h\" (UniqueName: \"kubernetes.io/projected/df762a3b-8602-4846-be8c-32ac8fd3b237-kube-api-access-hqn7h\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.329025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.328874 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-log-socket\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.329025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.328886 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-device-dir\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.329025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.328878 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-run-ovn\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.329025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.328921 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-run-ovn\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.329025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.328888 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-cni-bin\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.329025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.328941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz8gn\" (UniqueName: \"kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn\") pod \"network-check-target-jh8dz\" (UID: \"2a5fca6f-9523-47c5-b7f4-c6f45277ce47\") " pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:46:48.329025 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.328957 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 22:46:48.329025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.328964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-sys-fs\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.329025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.328990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.329025 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.329017 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret podName:0677a18f-b178-455f-a982-4c31ec52aab0 nodeName:}" failed. No retries permitted until 2026-07-08 22:46:48.828998411 +0000 UTC m=+3.084446093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret") pod "global-pull-secret-syncer-twk7h" (UID: "0677a18f-b178-455f-a982-4c31ec52aab0") : object "kube-system"/"original-pull-secret" not registered Jul 08 22:46:48.329025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329030 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329036 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-sys-fs\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329052 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63c5c2ac-8599-49be-b250-926ec607c77e-env-overrides\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329072 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c0a262a-d0f9-4fd3-91dc-44848d6d50b2-host\") pod \"node-ca-h7dmt\" (UID: \"9c0a262a-d0f9-4fd3-91dc-44848d6d50b2\") " pod="openshift-image-registry/node-ca-h7dmt" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-etc-selinux\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329133 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c0a262a-d0f9-4fd3-91dc-44848d6d50b2-host\") pod \"node-ca-h7dmt\" (UID: \"9c0a262a-d0f9-4fd3-91dc-44848d6d50b2\") " pod="openshift-image-registry/node-ca-h7dmt" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-node-log\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e72ff1d4-bb03-44e5-80cb-8f8002a894e8-konnectivity-ca\") pod \"konnectivity-agent-grqlf\" (UID: \"e72ff1d4-bb03-44e5-80cb-8f8002a894e8\") " pod="kube-system/konnectivity-agent-grqlf" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e72ff1d4-bb03-44e5-80cb-8f8002a894e8-agent-certs\") pod \"konnectivity-agent-grqlf\" (UID: \"e72ff1d4-bb03-44e5-80cb-8f8002a894e8\") " pod="kube-system/konnectivity-agent-grqlf" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329234 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-node-log\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329225 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0677a18f-b178-455f-a982-4c31ec52aab0-dbus\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329264 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-etc-selinux\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-registration-dir\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329314 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-cni-netd\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4m68\" (UniqueName: \"kubernetes.io/projected/63c5c2ac-8599-49be-b250-926ec607c77e-kube-api-access-q4m68\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.329713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329357 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0677a18f-b178-455f-a982-4c31ec52aab0-dbus\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:48.330230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3168d934-7d87-498f-9733-8059ff036149-system-cni-dir\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.330230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329383 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-registration-dir\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.330230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0677a18f-b178-455f-a982-4c31ec52aab0-kubelet-config\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:48.330230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329425 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-cni-netd\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329436 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-run-systemd\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329475 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-run-systemd\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329473 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0677a18f-b178-455f-a982-4c31ec52aab0-kubelet-config\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:48.330230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-etc-openvswitch\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329512 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-etc-openvswitch\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329431 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3168d934-7d87-498f-9733-8059ff036149-system-cni-dir\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.330230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-run-ovn-kubernetes\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329548 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-run-ovn-kubernetes\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329573 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63c5c2ac-8599-49be-b250-926ec607c77e-env-overrides\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329822 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e72ff1d4-bb03-44e5-80cb-8f8002a894e8-konnectivity-ca\") pod \"konnectivity-agent-grqlf\" (UID: \"e72ff1d4-bb03-44e5-80cb-8f8002a894e8\") " pod="kube-system/konnectivity-agent-grqlf" Jul 08 22:46:48.330714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.329566 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3168d934-7d87-498f-9733-8059ff036149-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.330714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c0a262a-d0f9-4fd3-91dc-44848d6d50b2-serviceca\") pod \"node-ca-h7dmt\" (UID: \"9c0a262a-d0f9-4fd3-91dc-44848d6d50b2\") " pod="openshift-image-registry/node-ca-h7dmt" Jul 08 22:46:48.330714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-kubelet\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-slash\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-run-openvswitch\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.330714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63c5c2ac-8599-49be-b250-926ec607c77e-ovnkube-script-lib\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3168d934-7d87-498f-9733-8059ff036149-cni-binary-copy\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.330714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330217 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-slash\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330252 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-run-openvswitch\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330267 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-kubelet\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.330714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330447 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.330714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmh7\" (UniqueName: \"kubernetes.io/projected/3168d934-7d87-498f-9733-8059ff036149-kube-api-access-ncmh7\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.330714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-run-netns\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.331240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330541 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63c5c2ac-8599-49be-b250-926ec607c77e-ovnkube-config\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.331240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3168d934-7d87-498f-9733-8059ff036149-os-release\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.331240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330597 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3168d934-7d87-498f-9733-8059ff036149-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.331240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330624 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-systemd-units\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.331240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330651 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3168d934-7d87-498f-9733-8059ff036149-cnibin\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.331240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330680 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3168d934-7d87-498f-9733-8059ff036149-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.331240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3168d934-7d87-498f-9733-8059ff036149-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.331240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330756 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-socket-dir\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.331240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330783 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3168d934-7d87-498f-9733-8059ff036149-cni-binary-copy\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.331240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330788 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-var-lib-openvswitch\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.331240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330824 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-var-lib-openvswitch\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.331240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330830 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c0a262a-d0f9-4fd3-91dc-44848d6d50b2-serviceca\") pod \"node-ca-h7dmt\" (UID: \"9c0a262a-d0f9-4fd3-91dc-44848d6d50b2\") " pod="openshift-image-registry/node-ca-h7dmt" Jul 08 22:46:48.331240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330715 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63c5c2ac-8599-49be-b250-926ec607c77e-ovnkube-script-lib\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.331240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63c5c2ac-8599-49be-b250-926ec607c77e-ovn-node-metrics-cert\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.331733 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330904 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmd9f\" (UniqueName: \"kubernetes.io/projected/9c0a262a-d0f9-4fd3-91dc-44848d6d50b2-kube-api-access-dmd9f\") pod \"node-ca-h7dmt\" (UID: \"9c0a262a-d0f9-4fd3-91dc-44848d6d50b2\") " pod="openshift-image-registry/node-ca-h7dmt" Jul 08 22:46:48.331733 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.330928 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df762a3b-8602-4846-be8c-32ac8fd3b237-socket-dir\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.331733 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.331069 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3168d934-7d87-498f-9733-8059ff036149-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.331733 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.331134 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3168d934-7d87-498f-9733-8059ff036149-os-release\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.331733 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.331187 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3168d934-7d87-498f-9733-8059ff036149-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.331733 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.331200 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-host-run-netns\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.331733 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.331191 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63c5c2ac-8599-49be-b250-926ec607c77e-ovnkube-config\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.331733 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.331231 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3168d934-7d87-498f-9733-8059ff036149-cnibin\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.331733 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.331233 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63c5c2ac-8599-49be-b250-926ec607c77e-systemd-units\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.332006 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.331930 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e72ff1d4-bb03-44e5-80cb-8f8002a894e8-agent-certs\") pod \"konnectivity-agent-grqlf\" (UID: \"e72ff1d4-bb03-44e5-80cb-8f8002a894e8\") " pod="kube-system/konnectivity-agent-grqlf" Jul 08 22:46:48.332958 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.332941 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63c5c2ac-8599-49be-b250-926ec607c77e-ovn-node-metrics-cert\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.339989 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.339971 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqn7h\" (UniqueName: \"kubernetes.io/projected/df762a3b-8602-4846-be8c-32ac8fd3b237-kube-api-access-hqn7h\") pod \"aws-ebs-csi-driver-node-k9bd5\" (UID: \"df762a3b-8602-4846-be8c-32ac8fd3b237\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.342228 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.341887 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 22:46:48.342228 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.341917 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 22:46:48.342228 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.341942 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sz8gn for pod openshift-network-diagnostics/network-check-target-jh8dz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 22:46:48.342228 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.342021 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn podName:2a5fca6f-9523-47c5-b7f4-c6f45277ce47 nodeName:}" failed. No retries permitted until 2026-07-08 22:46:48.841993 +0000 UTC m=+3.097440702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sz8gn" (UniqueName: "kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn") pod "network-check-target-jh8dz" (UID: "2a5fca6f-9523-47c5-b7f4-c6f45277ce47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 22:46:48.345642 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.345620 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmd9f\" (UniqueName: \"kubernetes.io/projected/9c0a262a-d0f9-4fd3-91dc-44848d6d50b2-kube-api-access-dmd9f\") pod \"node-ca-h7dmt\" (UID: \"9c0a262a-d0f9-4fd3-91dc-44848d6d50b2\") " pod="openshift-image-registry/node-ca-h7dmt" Jul 08 22:46:48.345745 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.345724 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4m68\" (UniqueName: \"kubernetes.io/projected/63c5c2ac-8599-49be-b250-926ec607c77e-kube-api-access-q4m68\") pod \"ovnkube-node-pd2mh\" (UID: \"63c5c2ac-8599-49be-b250-926ec607c77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.346444 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.346427 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmh7\" (UniqueName: \"kubernetes.io/projected/3168d934-7d87-498f-9733-8059ff036149-kube-api-access-ncmh7\") pod \"multus-additional-cni-plugins-gjw2b\" (UID: \"3168d934-7d87-498f-9733-8059ff036149\") " pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.413578 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.413525 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j6kcc" Jul 08 22:46:48.423499 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.423472 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qjtkc" Jul 08 22:46:48.425857 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:48.425834 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c226b53_16d4_436d_8cae_e37ab29dafeb.slice/crio-3b870ef04f17cee99dbd45c051fbcfd5c923424f73346d6d0ff9b195e9ce75c4 WatchSource:0}: Error finding container 3b870ef04f17cee99dbd45c051fbcfd5c923424f73346d6d0ff9b195e9ce75c4: Status 404 returned error can't find the container with id 3b870ef04f17cee99dbd45c051fbcfd5c923424f73346d6d0ff9b195e9ce75c4 Jul 08 22:46:48.430354 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.430332 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" Jul 08 22:46:48.435646 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.435584 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-297fk" Jul 08 22:46:48.440016 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:48.439989 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b588ae_74ca_487b_bc6b_8db2fb29691d.slice/crio-f1e240a9516f07042d4c3ec5607a145eb9696ddd54479f07716b2d090c0f0e83 WatchSource:0}: Error finding container f1e240a9516f07042d4c3ec5607a145eb9696ddd54479f07716b2d090c0f0e83: Status 404 returned error can't find the container with id f1e240a9516f07042d4c3ec5607a145eb9696ddd54479f07716b2d090c0f0e83 Jul 08 22:46:48.442815 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.442776 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gjw2b" Jul 08 22:46:48.446472 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:48.446445 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf82025_12e7_4d8d_a76a_127dd040d050.slice/crio-d38d4c3c32e7520979ad19788722f9b7961c4c8ea22627583da674d0b6a6e518 WatchSource:0}: Error finding container d38d4c3c32e7520979ad19788722f9b7961c4c8ea22627583da674d0b6a6e518: Status 404 returned error can't find the container with id d38d4c3c32e7520979ad19788722f9b7961c4c8ea22627583da674d0b6a6e518 Jul 08 22:46:48.448640 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.448620 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h7dmt" Jul 08 22:46:48.451614 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:48.451596 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3168d934_7d87_498f_9733_8059ff036149.slice/crio-9e285d30bf8b5612aa487a1d64738a5f1e74cf6ccf933f5250d378d131e1dd0e WatchSource:0}: Error finding container 9e285d30bf8b5612aa487a1d64738a5f1e74cf6ccf933f5250d378d131e1dd0e: Status 404 returned error can't find the container with id 9e285d30bf8b5612aa487a1d64738a5f1e74cf6ccf933f5250d378d131e1dd0e Jul 08 22:46:48.453900 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.453884 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-grqlf" Jul 08 22:46:48.457245 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:48.457226 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0a262a_d0f9_4fd3_91dc_44848d6d50b2.slice/crio-ef46628a419ce6e31b6daea501ce93f048c3efeb6d105e861a8b406b71a85b34 WatchSource:0}: Error finding container ef46628a419ce6e31b6daea501ce93f048c3efeb6d105e861a8b406b71a85b34: Status 404 returned error can't find the container with id ef46628a419ce6e31b6daea501ce93f048c3efeb6d105e861a8b406b71a85b34 Jul 08 22:46:48.459260 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.459242 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" Jul 08 22:46:48.462616 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:48.462596 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72ff1d4_bb03_44e5_80cb_8f8002a894e8.slice/crio-cc3416edc31c4c8f846746fd5c03fcd000992635f318f1eef5b1a1b17ebeca47 WatchSource:0}: Error finding container cc3416edc31c4c8f846746fd5c03fcd000992635f318f1eef5b1a1b17ebeca47: Status 404 returned error can't find the container with id cc3416edc31c4c8f846746fd5c03fcd000992635f318f1eef5b1a1b17ebeca47 Jul 08 22:46:48.465229 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.465121 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:46:48.468360 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:48.468335 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf762a3b_8602_4846_be8c_32ac8fd3b237.slice/crio-b7564e0467331a948312d226e89eaa26f847cf22989b3cb28a9411168d80a75d WatchSource:0}: Error finding container b7564e0467331a948312d226e89eaa26f847cf22989b3cb28a9411168d80a75d: Status 404 returned error can't find the container with id b7564e0467331a948312d226e89eaa26f847cf22989b3cb28a9411168d80a75d Jul 08 22:46:48.475325 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:46:48.475299 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c5c2ac_8599_49be_b250_926ec607c77e.slice/crio-a491100594a561f4fa46376ebf740d5dda4bdd81c436e3591b503f55f8bd4d11 WatchSource:0}: Error finding container a491100594a561f4fa46376ebf740d5dda4bdd81c436e3591b503f55f8bd4d11: Status 404 returned error can't find the container with id a491100594a561f4fa46376ebf740d5dda4bdd81c436e3591b503f55f8bd4d11 Jul 08 22:46:48.577384 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.577353 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 22:46:48.734378 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.734303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:48.734519 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.734427 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 22:46:48.734519 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.734503 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs podName:4ed6eae7-7b43-47ba-b489-136fab2a8ade nodeName:}" failed. No retries permitted until 2026-07-08 22:46:49.734464191 +0000 UTC m=+3.989911873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs") pod "network-metrics-daemon-rldqz" (UID: "4ed6eae7-7b43-47ba-b489-136fab2a8ade") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 22:46:48.834614 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.834585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:48.834796 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.834715 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 22:46:48.834796 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.834761 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret podName:0677a18f-b178-455f-a982-4c31ec52aab0 nodeName:}" failed. No retries permitted until 2026-07-08 22:46:49.834747179 +0000 UTC m=+4.090194867 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret") pod "global-pull-secret-syncer-twk7h" (UID: "0677a18f-b178-455f-a982-4c31ec52aab0") : object "kube-system"/"original-pull-secret" not registered Jul 08 22:46:48.935940 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:48.935908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz8gn\" (UniqueName: \"kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn\") pod \"network-check-target-jh8dz\" (UID: \"2a5fca6f-9523-47c5-b7f4-c6f45277ce47\") " pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:46:48.936200 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.936066 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 22:46:48.936200 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.936090 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 22:46:48.936200 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.936105 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sz8gn for pod openshift-network-diagnostics/network-check-target-jh8dz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 22:46:48.936200 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:48.936202 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn podName:2a5fca6f-9523-47c5-b7f4-c6f45277ce47 nodeName:}" failed. No retries permitted until 2026-07-08 22:46:49.936182838 +0000 UTC m=+4.191630524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sz8gn" (UniqueName: "kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn") pod "network-check-target-jh8dz" (UID: "2a5fca6f-9523-47c5-b7f4-c6f45277ce47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 22:46:49.163175 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.163046 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 22:41:48 +0000 UTC" deadline="2028-04-04 14:05:32.573767672 +0000 UTC" Jul 08 22:46:49.163175 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.163115 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15255h18m43.410657585s" Jul 08 22:46:49.263232 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.262409 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:49.263232 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.262416 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:46:49.263232 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:49.262552 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:46:49.263232 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:49.262581 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:46:49.316146 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.314944 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h7dmt" event={"ID":"9c0a262a-d0f9-4fd3-91dc-44848d6d50b2","Type":"ContainerStarted","Data":"ef46628a419ce6e31b6daea501ce93f048c3efeb6d105e861a8b406b71a85b34"} Jul 08 22:46:49.330051 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.329951 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-297fk" event={"ID":"eaf82025-12e7-4d8d-a76a-127dd040d050","Type":"ContainerStarted","Data":"d38d4c3c32e7520979ad19788722f9b7961c4c8ea22627583da674d0b6a6e518"} Jul 08 22:46:49.335933 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.335897 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j6kcc" event={"ID":"5c226b53-16d4-436d-8cae-e37ab29dafeb","Type":"ContainerStarted","Data":"3b870ef04f17cee99dbd45c051fbcfd5c923424f73346d6d0ff9b195e9ce75c4"} Jul 08 22:46:49.341271 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.341145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" event={"ID":"63c5c2ac-8599-49be-b250-926ec607c77e","Type":"ContainerStarted","Data":"a491100594a561f4fa46376ebf740d5dda4bdd81c436e3591b503f55f8bd4d11"} Jul 08 22:46:49.362333 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.362299 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" event={"ID":"df762a3b-8602-4846-be8c-32ac8fd3b237","Type":"ContainerStarted","Data":"b7564e0467331a948312d226e89eaa26f847cf22989b3cb28a9411168d80a75d"} Jul 08 22:46:49.380893 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.380859 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjw2b" event={"ID":"3168d934-7d87-498f-9733-8059ff036149","Type":"ContainerStarted","Data":"9e285d30bf8b5612aa487a1d64738a5f1e74cf6ccf933f5250d378d131e1dd0e"} Jul 08 22:46:49.397628 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.397590 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" event={"ID":"46b588ae-74ca-487b-bc6b-8db2fb29691d","Type":"ContainerStarted","Data":"f1e240a9516f07042d4c3ec5607a145eb9696ddd54479f07716b2d090c0f0e83"} Jul 08 22:46:49.414219 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.410546 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qjtkc" event={"ID":"150e8874-8d0a-4fb3-8079-bb87507a4393","Type":"ContainerStarted","Data":"1e0a4285d6d29a014da39c382bcd56db416d5e9c87f4c42b403b5ab36df25ba0"} Jul 08 22:46:49.427661 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.427595 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-grqlf" event={"ID":"e72ff1d4-bb03-44e5-80cb-8f8002a894e8","Type":"ContainerStarted","Data":"cc3416edc31c4c8f846746fd5c03fcd000992635f318f1eef5b1a1b17ebeca47"} Jul 08 22:46:49.742528 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.742429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:49.742721 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:49.742639 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 22:46:49.742721 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:49.742713 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs podName:4ed6eae7-7b43-47ba-b489-136fab2a8ade nodeName:}" failed. No retries permitted until 2026-07-08 22:46:51.74269047 +0000 UTC m=+5.998138157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs") pod "network-metrics-daemon-rldqz" (UID: "4ed6eae7-7b43-47ba-b489-136fab2a8ade") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 22:46:49.844056 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.843449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:49.844056 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:49.843609 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 22:46:49.844056 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:49.843681 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret podName:0677a18f-b178-455f-a982-4c31ec52aab0 nodeName:}" failed. No retries permitted until 2026-07-08 22:46:51.843657876 +0000 UTC m=+6.099105573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret") pod "global-pull-secret-syncer-twk7h" (UID: "0677a18f-b178-455f-a982-4c31ec52aab0") : object "kube-system"/"original-pull-secret" not registered Jul 08 22:46:49.944000 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:49.943963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz8gn\" (UniqueName: \"kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn\") pod \"network-check-target-jh8dz\" (UID: \"2a5fca6f-9523-47c5-b7f4-c6f45277ce47\") " pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:46:49.944223 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:49.944191 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 22:46:49.944223 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:49.944212 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 22:46:49.944342 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:49.944225 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sz8gn for pod openshift-network-diagnostics/network-check-target-jh8dz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 22:46:49.944342 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:49.944288 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn podName:2a5fca6f-9523-47c5-b7f4-c6f45277ce47 nodeName:}" failed. No retries permitted until 2026-07-08 22:46:51.944266892 +0000 UTC m=+6.199714583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sz8gn" (UniqueName: "kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn") pod "network-check-target-jh8dz" (UID: "2a5fca6f-9523-47c5-b7f4-c6f45277ce47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 22:46:50.163784 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:50.163682 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 22:41:48 +0000 UTC" deadline="2028-03-04 05:37:54.374017389 +0000 UTC" Jul 08 22:46:50.163784 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:50.163725 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14502h51m4.210296349s" Jul 08 22:46:50.265523 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:50.265488 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:50.265702 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:50.265628 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:46:51.261706 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:51.261668 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:51.262256 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:51.261833 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:46:51.262331 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:51.262277 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:46:51.262489 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:51.262383 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:46:51.761879 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:51.761205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:51.761879 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:51.761403 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 22:46:51.761879 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:51.761470 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs podName:4ed6eae7-7b43-47ba-b489-136fab2a8ade nodeName:}" failed. No retries permitted until 2026-07-08 22:46:55.761450406 +0000 UTC m=+10.016898093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs") pod "network-metrics-daemon-rldqz" (UID: "4ed6eae7-7b43-47ba-b489-136fab2a8ade") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 22:46:51.862466 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:51.862423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:51.862645 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:51.862592 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 22:46:51.862724 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:51.862662 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret podName:0677a18f-b178-455f-a982-4c31ec52aab0 nodeName:}" failed. No retries permitted until 2026-07-08 22:46:55.862641853 +0000 UTC m=+10.118089547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret") pod "global-pull-secret-syncer-twk7h" (UID: "0677a18f-b178-455f-a982-4c31ec52aab0") : object "kube-system"/"original-pull-secret" not registered Jul 08 22:46:51.962942 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:51.962907 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz8gn\" (UniqueName: \"kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn\") pod \"network-check-target-jh8dz\" (UID: \"2a5fca6f-9523-47c5-b7f4-c6f45277ce47\") " pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:46:51.963136 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:51.963076 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 22:46:51.963136 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:51.963118 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 22:46:51.963136 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:51.963133 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sz8gn for pod openshift-network-diagnostics/network-check-target-jh8dz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 22:46:51.963293 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:51.963199 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn podName:2a5fca6f-9523-47c5-b7f4-c6f45277ce47 nodeName:}" failed. No retries permitted until 2026-07-08 22:46:55.963171278 +0000 UTC m=+10.218618966 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sz8gn" (UniqueName: "kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn") pod "network-check-target-jh8dz" (UID: "2a5fca6f-9523-47c5-b7f4-c6f45277ce47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 22:46:52.261874 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:52.261632 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:52.261874 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:52.261764 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:46:53.262249 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:53.262205 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:46:53.262772 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:53.262353 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:46:53.262772 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:53.262440 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:53.262772 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:53.262581 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:46:54.261919 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:54.261878 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:54.262104 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:54.262008 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:46:55.262380 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:55.262345 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:55.262843 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:55.262390 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:46:55.262843 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:55.262494 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:46:55.262843 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:55.262598 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:46:55.800347 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:55.800301 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:55.800523 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:55.800483 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 22:46:55.800595 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:55.800553 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs podName:4ed6eae7-7b43-47ba-b489-136fab2a8ade nodeName:}" failed. No retries permitted until 2026-07-08 22:47:03.800534329 +0000 UTC m=+18.055982016 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs") pod "network-metrics-daemon-rldqz" (UID: "4ed6eae7-7b43-47ba-b489-136fab2a8ade") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 22:46:55.901263 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:55.901214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:55.901433 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:55.901376 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 22:46:55.901502 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:55.901450 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret podName:0677a18f-b178-455f-a982-4c31ec52aab0 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:03.901430341 +0000 UTC m=+18.156878026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret") pod "global-pull-secret-syncer-twk7h" (UID: "0677a18f-b178-455f-a982-4c31ec52aab0") : object "kube-system"/"original-pull-secret" not registered Jul 08 22:46:56.002569 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:56.002527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz8gn\" (UniqueName: \"kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn\") pod \"network-check-target-jh8dz\" (UID: \"2a5fca6f-9523-47c5-b7f4-c6f45277ce47\") " pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:46:56.002772 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:56.002753 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 22:46:56.002853 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:56.002783 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 22:46:56.002853 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:56.002798 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sz8gn for pod openshift-network-diagnostics/network-check-target-jh8dz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 22:46:56.002969 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:56.002857 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn podName:2a5fca6f-9523-47c5-b7f4-c6f45277ce47 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:04.002838379 +0000 UTC m=+18.258286075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sz8gn" (UniqueName: "kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn") pod "network-check-target-jh8dz" (UID: "2a5fca6f-9523-47c5-b7f4-c6f45277ce47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 22:46:56.262912 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:56.262428 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:56.262912 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:56.262542 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:46:57.261341 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:57.261272 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:57.261341 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:57.261323 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:46:57.261586 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:57.261407 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:46:57.261586 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:57.261543 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:46:58.262143 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:58.261694 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:46:58.262143 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:58.261833 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:46:59.261978 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:59.261943 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:46:59.262172 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:46:59.261998 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:46:59.262172 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:59.262112 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:46:59.262555 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:46:59.262200 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:47:00.262173 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:00.262140 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:00.262549 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:00.262242 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:47:01.261950 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:01.261917 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:01.262143 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:01.262059 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:47:01.262143 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:01.262100 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:01.262557 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:01.262207 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:47:02.261657 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:02.261624 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:02.261834 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:02.261760 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:47:03.261666 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:03.261638 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:03.262110 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:03.261758 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:47:03.262110 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:03.261820 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:03.262110 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:03.261919 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:47:03.859172 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:03.859134 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:03.859355 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:03.859328 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 22:47:03.859431 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:03.859416 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs podName:4ed6eae7-7b43-47ba-b489-136fab2a8ade nodeName:}" failed. No retries permitted until 2026-07-08 22:47:19.859393621 +0000 UTC m=+34.114841306 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs") pod "network-metrics-daemon-rldqz" (UID: "4ed6eae7-7b43-47ba-b489-136fab2a8ade") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 22:47:03.959734 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:03.959698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:03.959913 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:03.959846 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 22:47:03.959997 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:03.959915 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret podName:0677a18f-b178-455f-a982-4c31ec52aab0 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:19.959896598 +0000 UTC m=+34.215344291 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret") pod "global-pull-secret-syncer-twk7h" (UID: "0677a18f-b178-455f-a982-4c31ec52aab0") : object "kube-system"/"original-pull-secret" not registered Jul 08 22:47:04.060114 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:04.060065 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz8gn\" (UniqueName: \"kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn\") pod \"network-check-target-jh8dz\" (UID: \"2a5fca6f-9523-47c5-b7f4-c6f45277ce47\") " pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:04.060288 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:04.060256 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 22:47:04.060288 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:04.060281 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 22:47:04.060399 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:04.060294 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sz8gn for pod openshift-network-diagnostics/network-check-target-jh8dz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 22:47:04.060399 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:04.060360 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn podName:2a5fca6f-9523-47c5-b7f4-c6f45277ce47 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:20.060341196 +0000 UTC m=+34.315788891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sz8gn" (UniqueName: "kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn") pod "network-check-target-jh8dz" (UID: "2a5fca6f-9523-47c5-b7f4-c6f45277ce47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 22:47:04.261623 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:04.261555 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:04.261757 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:04.261673 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:47:05.261572 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:05.261533 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:05.261763 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:05.261668 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:47:05.261763 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:05.261725 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:05.262167 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:05.261838 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:47:06.263267 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:06.263225 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:06.263716 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:06.263346 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:47:07.261980 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:07.261946 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:07.262172 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:07.261946 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:07.262172 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:07.262095 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:47:07.262172 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:07.262163 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:47:08.261703 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:08.261663 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:08.262155 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:08.261810 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:47:09.261645 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:09.261610 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:09.261866 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:09.261727 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:47:09.261866 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:09.261781 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:09.262316 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:09.261899 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:47:09.473194 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:09.473114 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" event={"ID":"be9286b9c4dca1e8d98a40958054271d","Type":"ContainerStarted","Data":"67c5e0c1d913c67063fd13425e0f6fa64c8935316adbad99ec4d1ade8318491b"} Jul 08 22:47:09.474679 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:09.474652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j6kcc" event={"ID":"5c226b53-16d4-436d-8cae-e37ab29dafeb","Type":"ContainerStarted","Data":"8ce38c6225284bdbd47df3aa4e41f119a4e5c7f10677daa8db592fd3efd45d72"} Jul 08 22:47:09.478077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:09.478055 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" event={"ID":"63c5c2ac-8599-49be-b250-926ec607c77e","Type":"ContainerStarted","Data":"4a056aceaf4eca7a3a949057083a80465110c815d78a7c7bd495b3f5d3899890"} Jul 08 22:47:09.479437 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:09.479415 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" event={"ID":"46b588ae-74ca-487b-bc6b-8db2fb29691d","Type":"ContainerStarted","Data":"b3ead6d53e7cdcab2e252150f94c3124cd137caf289b5bf78502a5db94d570da"} Jul 08 22:47:09.491941 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:09.491905 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" podStartSLOduration=21.491893469 podStartE2EDuration="21.491893469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 22:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:47:09.491836686 +0000 UTC m=+23.747284593" watchObservedRunningTime="2026-07-08 22:47:09.491893469 +0000 UTC m=+23.747341172" Jul 08 22:47:09.509605 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:09.509430 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j6kcc" podStartSLOduration=3.200422911 podStartE2EDuration="23.509415179s" totalImagesPullingTime="20.308992268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 22:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:47:09.508873585 +0000 UTC m=+23.764321289" watchObservedRunningTime="2026-07-08 22:47:09.509415179 +0000 UTC m=+23.764862883" Jul 08 22:47:09.527263 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:09.527220 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9t6cr" podStartSLOduration=3.250814265 podStartE2EDuration="23.527201728s" totalImagesPullingTime="20.276387463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 22:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:47:09.527020526 +0000 UTC m=+23.782468232" watchObservedRunningTime="2026-07-08 22:47:09.527201728 +0000 UTC m=+23.782649433" Jul 08 22:47:10.262460 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.262173 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:10.263111 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:10.262532 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:47:10.486414 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.486387 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 22:47:10.486746 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.486724 2564 generic.go:356] "Generic (PLEG): container finished" podID="63c5c2ac-8599-49be-b250-926ec607c77e" containerID="6998f5ecc19e418cb26ab748cf1826a36c79f79642367ffdadaacf1a4a97392f" exitCode=1 Jul 08 22:47:10.486838 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.486795 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" event={"ID":"63c5c2ac-8599-49be-b250-926ec607c77e","Type":"ContainerStarted","Data":"029809374f5cdeb888be528987dab976b7bde6ad1ddd7dcc9e690d82771e3cc5"} Jul 08 22:47:10.486896 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.486876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" event={"ID":"63c5c2ac-8599-49be-b250-926ec607c77e","Type":"ContainerStarted","Data":"cc374c4889b87a1303f8088d0fcf0359ae400618cf4317a9a855fbaac248fac8"} Jul 08 22:47:10.486941 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.486907 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" event={"ID":"63c5c2ac-8599-49be-b250-926ec607c77e","Type":"ContainerStarted","Data":"ca3a17cae2515a720b768e79ea0f586e89cdbaa4e10e1d6dcfc40f1d5abcfbaf"} Jul 08 22:47:10.486975 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.486941 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" event={"ID":"63c5c2ac-8599-49be-b250-926ec607c77e","Type":"ContainerStarted","Data":"9476da3a578b22aaa64f4796a43a42be9f902bb2b860d12d05115f1d9a87240c"} Jul 08 22:47:10.487020 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.486976 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" event={"ID":"63c5c2ac-8599-49be-b250-926ec607c77e","Type":"ContainerDied","Data":"6998f5ecc19e418cb26ab748cf1826a36c79f79642367ffdadaacf1a4a97392f"} Jul 08 22:47:10.488255 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.488233 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" event={"ID":"df762a3b-8602-4846-be8c-32ac8fd3b237","Type":"ContainerStarted","Data":"853e37a5e097891f175e58cac28fdec4057a01307ae0f32538ca11bf2fc9b14f"} Jul 08 22:47:10.489700 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.489681 2564 generic.go:356] "Generic (PLEG): container finished" podID="3168d934-7d87-498f-9733-8059ff036149" containerID="667a51032c91326a42de4856267482ce4c00f6f7cf9f106cc15efde38c53cb26" exitCode=0 Jul 08 22:47:10.489778 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.489746 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjw2b" event={"ID":"3168d934-7d87-498f-9733-8059ff036149","Type":"ContainerDied","Data":"667a51032c91326a42de4856267482ce4c00f6f7cf9f106cc15efde38c53cb26"} Jul 08 22:47:10.491264 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.491218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qjtkc" event={"ID":"150e8874-8d0a-4fb3-8079-bb87507a4393","Type":"ContainerStarted","Data":"0976b4aa98c7938f1951094dfbfba5cf48812ce1f8275fad37369165cdcf942a"} Jul 08 22:47:10.494675 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.492780 2564 generic.go:356] "Generic (PLEG): container finished" podID="1f8a59a8baac1caed8261311e43cf1fc" containerID="778170b57b67f2305cb1a06942a0c57be3ef0ce8f705cc776db3d2f8d53e9932" exitCode=0 Jul 08 22:47:10.494675 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.492856 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" event={"ID":"1f8a59a8baac1caed8261311e43cf1fc","Type":"ContainerDied","Data":"778170b57b67f2305cb1a06942a0c57be3ef0ce8f705cc776db3d2f8d53e9932"} Jul 08 22:47:10.496857 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.496838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-grqlf" event={"ID":"e72ff1d4-bb03-44e5-80cb-8f8002a894e8","Type":"ContainerStarted","Data":"b7a8bce4c9085331a1a8c20308e8664a243114aed5ac81b479faaf4e9bab4027"} Jul 08 22:47:10.498153 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.498136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h7dmt" event={"ID":"9c0a262a-d0f9-4fd3-91dc-44848d6d50b2","Type":"ContainerStarted","Data":"ca42b27074b8f7c3498bbafadae2b00af5b5a5de7735c95844666669cf172896"} Jul 08 22:47:10.499438 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.499416 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-297fk" event={"ID":"eaf82025-12e7-4d8d-a76a-127dd040d050","Type":"ContainerStarted","Data":"8416b379e91eb626c5e072277a90b74508c8eb8ba9ad36cb6da45f58486e3145"} Jul 08 22:47:10.524181 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.524145 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qjtkc" podStartSLOduration=4.25942894 podStartE2EDuration="24.524132983s" totalImagesPullingTime="20.264704043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 22:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:47:10.523794389 +0000 UTC m=+24.779242094" watchObservedRunningTime="2026-07-08 22:47:10.524132983 +0000 UTC m=+24.779580687" Jul 08 22:47:10.537797 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.537752 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-297fk" podStartSLOduration=4.28652345 podStartE2EDuration="24.537737315s" totalImagesPullingTime="20.251213865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 22:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:47:10.536938251 +0000 UTC m=+24.792385954" watchObservedRunningTime="2026-07-08 22:47:10.537737315 +0000 UTC m=+24.793185020" Jul 08 22:47:10.563062 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.563026 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h7dmt" podStartSLOduration=4.304594891 podStartE2EDuration="24.563012368s" totalImagesPullingTime="20.258417477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 22:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:47:10.562799389 +0000 UTC m=+24.818247091" watchObservedRunningTime="2026-07-08 22:47:10.563012368 +0000 UTC m=+24.818460070" Jul 08 22:47:10.575165 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:10.575126 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-grqlf" podStartSLOduration=7.950318166 podStartE2EDuration="23.575111881s" totalImagesPullingTime="15.624793715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:47:10.574945881 +0000 UTC m=+24.830393584" watchObservedRunningTime="2026-07-08 22:47:10.575111881 +0000 UTC m=+24.830559583" Jul 08 22:47:11.261589 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:11.261554 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:11.261847 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:11.261606 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:11.261847 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:11.261695 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:47:11.261847 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:11.261821 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:47:11.503721 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:11.503682 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" event={"ID":"1f8a59a8baac1caed8261311e43cf1fc","Type":"ContainerStarted","Data":"f9cc4ffb512d301f24931b830f000fa40046cda0b61fb67c1de8d447c24b8dbd"} Jul 08 22:47:11.518075 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:11.517988 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" podStartSLOduration=23.517970316 podStartE2EDuration="23.517970316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 22:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:47:11.517647668 +0000 UTC m=+25.773095372" watchObservedRunningTime="2026-07-08 22:47:11.517970316 +0000 UTC m=+25.773418018" Jul 08 22:47:11.581516 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:11.581488 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 22:47:12.201425 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:12.201285 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T22:47:11.581508251Z","UUID":"e667af17-cd99-448c-8c29-eb0d199a816a","Handler":null,"Name":"","Endpoint":""} Jul 08 22:47:12.203544 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:12.203323 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 22:47:12.203544 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:12.203371 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 22:47:12.262561 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:12.262493 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:12.262713 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:12.262695 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:47:12.423713 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:12.423677 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-grqlf" Jul 08 22:47:12.424279 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:12.424260 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-grqlf" Jul 08 22:47:12.509349 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:12.509319 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" event={"ID":"df762a3b-8602-4846-be8c-32ac8fd3b237","Type":"ContainerStarted","Data":"89e9f051145987e3eb0895c629c5d056a4c4a64a2db91270ee8b3d12bb2eb2ce"} Jul 08 22:47:12.512320 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:12.512299 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 22:47:12.512673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:12.512630 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" event={"ID":"63c5c2ac-8599-49be-b250-926ec607c77e","Type":"ContainerStarted","Data":"a87aaa0c1ff43d92fd46080b6ba6aa1a17ac4b917b234164c304a45fa6aa37b9"} Jul 08 22:47:12.513049 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:12.513035 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-grqlf" Jul 08 22:47:12.513374 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:12.513360 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-grqlf" Jul 08 22:47:13.261558 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:13.261467 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:13.261735 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:13.261461 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:13.261735 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:13.261589 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:47:13.261825 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:13.261731 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:47:13.517149 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:13.517032 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" event={"ID":"df762a3b-8602-4846-be8c-32ac8fd3b237","Type":"ContainerStarted","Data":"6a4fd006334bbd8338b0e7b72b16dea04dc60f0af112d65a6d7f27158d59cb23"} Jul 08 22:47:13.534820 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:13.534778 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9bd5" podStartSLOduration=3.627762072 podStartE2EDuration="26.534763186s" totalImagesPullingTime="22.907001114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 22:46:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:47:13.533515634 +0000 UTC m=+27.788963336" watchObservedRunningTime="2026-07-08 22:47:13.534763186 +0000 UTC m=+27.790210888" Jul 08 22:47:14.262437 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:14.262320 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:14.262583 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:14.262545 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:47:14.526924 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:14.524472 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 22:47:14.527678 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:14.527339 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" event={"ID":"63c5c2ac-8599-49be-b250-926ec607c77e","Type":"ContainerStarted","Data":"bcd25b4be8ca52c85301ee30af838fd8ecb845097ee16715253f160c5c38413e"} Jul 08 22:47:14.527988 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:14.527966 2564 scope.go:117] "RemoveContainer" containerID="6998f5ecc19e418cb26ab748cf1826a36c79f79642367ffdadaacf1a4a97392f" Jul 08 22:47:15.261985 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:15.261949 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:15.262192 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:15.261949 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:15.262192 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:15.262095 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:47:15.262192 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:15.262166 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:47:16.118218 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.117992 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-twk7h"] Jul 08 22:47:16.118663 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.118305 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:16.118663 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:16.118406 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:47:16.121371 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.121341 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jh8dz"] Jul 08 22:47:16.121492 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.121455 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:16.121568 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:16.121553 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:47:16.122187 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.122165 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rldqz"] Jul 08 22:47:16.122278 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.122266 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:16.122400 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:16.122378 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:47:16.538511 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.538482 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 22:47:16.538890 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.538864 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" event={"ID":"63c5c2ac-8599-49be-b250-926ec607c77e","Type":"ContainerStarted","Data":"59d0f96af7a9890e849d3958a69effec314ba7d69aa5db66b9162cb36c5a9111"} Jul 08 22:47:16.539103 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.539037 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:47:16.539202 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.539124 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:47:16.539202 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.539175 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:47:16.540914 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.540884 2564 generic.go:356] "Generic (PLEG): container finished" podID="3168d934-7d87-498f-9733-8059ff036149" containerID="385c34e1ed49987d4828ba259275eb915a8295c96344aee571a3c07124d65cc3" exitCode=0 Jul 08 22:47:16.540999 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.540925 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjw2b" event={"ID":"3168d934-7d87-498f-9733-8059ff036149","Type":"ContainerDied","Data":"385c34e1ed49987d4828ba259275eb915a8295c96344aee571a3c07124d65cc3"} Jul 08 22:47:16.554577 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.554554 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:47:16.554681 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.554660 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:47:16.564674 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:16.564637 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" podStartSLOduration=8.772535969 podStartE2EDuration="29.564625892s" totalImagesPullingTime="20.792089923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 22:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:47:16.564149283 +0000 UTC m=+30.819596986" watchObservedRunningTime="2026-07-08 22:47:16.564625892 +0000 UTC m=+30.820073595" Jul 08 22:47:17.262347 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:17.262318 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:17.262765 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:17.262323 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:17.262765 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:17.262422 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:47:17.262765 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:17.262527 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:47:17.544922 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:17.544836 2564 generic.go:356] "Generic (PLEG): container finished" podID="3168d934-7d87-498f-9733-8059ff036149" containerID="cb86fd7add3940507996437f484e69ca7639336b88c2ba60a576360617b81a87" exitCode=0 Jul 08 22:47:17.545075 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:17.544923 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjw2b" event={"ID":"3168d934-7d87-498f-9733-8059ff036149","Type":"ContainerDied","Data":"cb86fd7add3940507996437f484e69ca7639336b88c2ba60a576360617b81a87"} Jul 08 22:47:18.264682 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:18.264649 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:18.265116 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:18.264764 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:47:18.549663 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:18.549572 2564 generic.go:356] "Generic (PLEG): container finished" podID="3168d934-7d87-498f-9733-8059ff036149" containerID="b5a5f176b963d3cf218ea351639412bc9f428adbe4d10125a6a4b5d2ec5d8638" exitCode=0 Jul 08 22:47:18.549818 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:18.549651 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjw2b" event={"ID":"3168d934-7d87-498f-9733-8059ff036149","Type":"ContainerDied","Data":"b5a5f176b963d3cf218ea351639412bc9f428adbe4d10125a6a4b5d2ec5d8638"} Jul 08 22:47:19.261789 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:19.261750 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:19.261789 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:19.261786 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:19.262037 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:19.261879 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:47:19.262037 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:19.261997 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:47:19.868777 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:19.868738 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:19.869296 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:19.868896 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 22:47:19.869296 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:19.868969 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs podName:4ed6eae7-7b43-47ba-b489-136fab2a8ade nodeName:}" failed. No retries permitted until 2026-07-08 22:47:51.868947403 +0000 UTC m=+66.124395090 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs") pod "network-metrics-daemon-rldqz" (UID: "4ed6eae7-7b43-47ba-b489-136fab2a8ade") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 22:47:19.969056 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:19.969017 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:19.969203 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:19.969169 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 22:47:19.969248 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:19.969229 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret podName:0677a18f-b178-455f-a982-4c31ec52aab0 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:51.96921449 +0000 UTC m=+66.224662175 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret") pod "global-pull-secret-syncer-twk7h" (UID: "0677a18f-b178-455f-a982-4c31ec52aab0") : object "kube-system"/"original-pull-secret" not registered Jul 08 22:47:20.070396 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:20.070363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz8gn\" (UniqueName: \"kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn\") pod \"network-check-target-jh8dz\" (UID: \"2a5fca6f-9523-47c5-b7f4-c6f45277ce47\") " pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:20.070550 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:20.070535 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 22:47:20.070597 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:20.070557 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 22:47:20.070597 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:20.070570 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sz8gn for pod openshift-network-diagnostics/network-check-target-jh8dz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 22:47:20.070690 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:20.070631 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn podName:2a5fca6f-9523-47c5-b7f4-c6f45277ce47 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:52.070615639 +0000 UTC m=+66.326063337 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sz8gn" (UniqueName: "kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn") pod "network-check-target-jh8dz" (UID: "2a5fca6f-9523-47c5-b7f4-c6f45277ce47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 22:47:20.261923 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:20.261847 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:20.262064 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:20.261947 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh8dz" podUID="2a5fca6f-9523-47c5-b7f4-c6f45277ce47" Jul 08 22:47:21.262160 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:21.262131 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:21.262522 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:21.262249 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:47:21.262522 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:21.262302 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:21.262522 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:21.262400 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twk7h" podUID="0677a18f-b178-455f-a982-4c31ec52aab0" Jul 08 22:47:22.062434 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.062402 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeReady" Jul 08 22:47:22.062631 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.062565 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 22:47:22.093373 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.093344 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-594fb948ff-jktv6"] Jul 08 22:47:22.127206 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.127173 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-pm98b"] Jul 08 22:47:22.127382 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.127288 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.129817 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.129780 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 08 22:47:22.129817 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.129790 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 08 22:47:22.130075 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.129822 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 08 22:47:22.130075 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.129843 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cvqzh\"" Jul 08 22:47:22.136498 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.136476 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 08 22:47:22.138752 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.138729 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jpnsr"] Jul 08 22:47:22.138887 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.138868 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:47:22.141226 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.141207 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 08 22:47:22.141324 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.141238 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-r8m6j\"" Jul 08 22:47:22.141324 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.141211 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 08 22:47:22.155242 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.155211 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-pm98b"] Jul 08 22:47:22.155346 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.155250 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-594fb948ff-jktv6"] Jul 08 22:47:22.155346 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.155274 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xv6dw"] Jul 08 22:47:22.155346 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.155295 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jpnsr" Jul 08 22:47:22.158583 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.158341 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 22:47:22.158583 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.158345 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 22:47:22.158583 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.158429 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c5p5l\"" Jul 08 22:47:22.159099 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.159071 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 22:47:22.169619 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.169601 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jpnsr"] Jul 08 22:47:22.169723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.169628 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xv6dw"] Jul 08 22:47:22.169773 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.169721 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:22.172127 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.172107 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 22:47:22.172240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.172186 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fbpb9\"" Jul 08 22:47:22.172240 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.172111 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 22:47:22.242180 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.242118 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:22.261889 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.261859 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:22.264583 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.264562 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 22:47:22.264583 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.264573 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7tfqh\"" Jul 08 22:47:22.264978 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.264602 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 22:47:22.285439 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285413 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/407a5c0c-24fd-474d-b677-65c32d5c1ca5-tmp-dir\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:22.285569 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285451 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-bound-sa-token\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.285569 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285480 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:22.285666 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285581 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2d8cc6ed-a2e9-422d-ac36-2970b983252a-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:47:22.285666 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82dddf27-89f2-45f8-ba41-74a7be616df4-installation-pull-secrets\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.285666 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285650 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/407a5c0c-24fd-474d-b677-65c32d5c1ca5-config-volume\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:22.285809 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285684 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:47:22.285809 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285707 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32679a71-0c9d-4b6a-8b60-130bd59ae45c-cert\") pod \"ingress-canary-jpnsr\" (UID: \"32679a71-0c9d-4b6a-8b60-130bd59ae45c\") " pod="openshift-ingress-canary/ingress-canary-jpnsr" Jul 08 22:47:22.285809 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285795 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.285958 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285835 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs22h\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-kube-api-access-bs22h\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.285958 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285867 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5ql\" (UniqueName: \"kubernetes.io/projected/407a5c0c-24fd-474d-b677-65c32d5c1ca5-kube-api-access-6f5ql\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:22.285958 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285897 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5c6\" (UniqueName: \"kubernetes.io/projected/32679a71-0c9d-4b6a-8b60-130bd59ae45c-kube-api-access-bp5c6\") pod \"ingress-canary-jpnsr\" (UID: \"32679a71-0c9d-4b6a-8b60-130bd59ae45c\") " pod="openshift-ingress-canary/ingress-canary-jpnsr" Jul 08 22:47:22.285958 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285925 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82dddf27-89f2-45f8-ba41-74a7be616df4-image-registry-private-configuration\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.285958 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285954 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82dddf27-89f2-45f8-ba41-74a7be616df4-ca-trust-extracted\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.286205 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.285986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-certificates\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.286205 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.286013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82dddf27-89f2-45f8-ba41-74a7be616df4-trusted-ca\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.320381 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.320307 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:22.387307 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5ql\" (UniqueName: \"kubernetes.io/projected/407a5c0c-24fd-474d-b677-65c32d5c1ca5-kube-api-access-6f5ql\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:22.387307 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5c6\" (UniqueName: \"kubernetes.io/projected/32679a71-0c9d-4b6a-8b60-130bd59ae45c-kube-api-access-bp5c6\") pod \"ingress-canary-jpnsr\" (UID: \"32679a71-0c9d-4b6a-8b60-130bd59ae45c\") " pod="openshift-ingress-canary/ingress-canary-jpnsr" Jul 08 22:47:22.387548 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82dddf27-89f2-45f8-ba41-74a7be616df4-image-registry-private-configuration\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.387548 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82dddf27-89f2-45f8-ba41-74a7be616df4-ca-trust-extracted\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.387548 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-certificates\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.387548 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82dddf27-89f2-45f8-ba41-74a7be616df4-trusted-ca\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.387548 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387445 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/407a5c0c-24fd-474d-b677-65c32d5c1ca5-tmp-dir\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:22.387832 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387562 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-bound-sa-token\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.387832 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:22.387832 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387653 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2d8cc6ed-a2e9-422d-ac36-2970b983252a-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:47:22.387832 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82dddf27-89f2-45f8-ba41-74a7be616df4-installation-pull-secrets\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.387832 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/407a5c0c-24fd-474d-b677-65c32d5c1ca5-config-volume\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:22.387832 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:47:22.387832 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387782 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32679a71-0c9d-4b6a-8b60-130bd59ae45c-cert\") pod \"ingress-canary-jpnsr\" (UID: \"32679a71-0c9d-4b6a-8b60-130bd59ae45c\") " pod="openshift-ingress-canary/ingress-canary-jpnsr" Jul 08 22:47:22.387832 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:22.387790 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 22:47:22.387832 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.388370 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387838 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82dddf27-89f2-45f8-ba41-74a7be616df4-ca-trust-extracted\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.388370 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:22.387853 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls podName:407a5c0c-24fd-474d-b677-65c32d5c1ca5 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:22.887831866 +0000 UTC m=+37.143279547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls") pod "dns-default-xv6dw" (UID: "407a5c0c-24fd-474d-b677-65c32d5c1ca5") : secret "dns-default-metrics-tls" not found Jul 08 22:47:22.388370 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs22h\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-kube-api-access-bs22h\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.388370 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:22.387888 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 22:47:22.388370 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:22.387902 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-594fb948ff-jktv6: secret "image-registry-tls" not found Jul 08 22:47:22.388370 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:22.387965 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls podName:82dddf27-89f2-45f8-ba41-74a7be616df4 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:22.887950836 +0000 UTC m=+37.143398536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls") pod "image-registry-594fb948ff-jktv6" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4") : secret "image-registry-tls" not found Jul 08 22:47:22.388370 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:22.388047 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 22:47:22.388370 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:22.388126 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert podName:2d8cc6ed-a2e9-422d-ac36-2970b983252a nodeName:}" failed. No retries permitted until 2026-07-08 22:47:22.888111973 +0000 UTC m=+37.143559663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pm98b" (UID: "2d8cc6ed-a2e9-422d-ac36-2970b983252a") : secret "networking-console-plugin-cert" not found Jul 08 22:47:22.388370 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.387781 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/407a5c0c-24fd-474d-b677-65c32d5c1ca5-tmp-dir\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:22.388370 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.388170 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-certificates\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.388860 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.388538 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82dddf27-89f2-45f8-ba41-74a7be616df4-trusted-ca\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.388860 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.388606 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2d8cc6ed-a2e9-422d-ac36-2970b983252a-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:47:22.388860 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.388709 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/407a5c0c-24fd-474d-b677-65c32d5c1ca5-config-volume\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:22.391821 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.391794 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82dddf27-89f2-45f8-ba41-74a7be616df4-image-registry-private-configuration\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.391821 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.391813 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82dddf27-89f2-45f8-ba41-74a7be616df4-installation-pull-secrets\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.391959 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.391824 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32679a71-0c9d-4b6a-8b60-130bd59ae45c-cert\") pod \"ingress-canary-jpnsr\" (UID: \"32679a71-0c9d-4b6a-8b60-130bd59ae45c\") " pod="openshift-ingress-canary/ingress-canary-jpnsr" Jul 08 22:47:22.395765 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.395741 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-bound-sa-token\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.396050 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.396026 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5c6\" (UniqueName: \"kubernetes.io/projected/32679a71-0c9d-4b6a-8b60-130bd59ae45c-kube-api-access-bp5c6\") pod \"ingress-canary-jpnsr\" (UID: \"32679a71-0c9d-4b6a-8b60-130bd59ae45c\") " pod="openshift-ingress-canary/ingress-canary-jpnsr" Jul 08 22:47:22.396206 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.396185 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs22h\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-kube-api-access-bs22h\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.396388 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.396364 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5ql\" (UniqueName: \"kubernetes.io/projected/407a5c0c-24fd-474d-b677-65c32d5c1ca5-kube-api-access-6f5ql\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:22.465626 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.465594 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jpnsr" Jul 08 22:47:22.665939 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.665760 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jpnsr"] Jul 08 22:47:22.670363 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:47:22.670324 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32679a71_0c9d_4b6a_8b60_130bd59ae45c.slice/crio-91786209211b4a0aac050271ce78de6974ca604e1883cedec26a2052f06d5889 WatchSource:0}: Error finding container 91786209211b4a0aac050271ce78de6974ca604e1883cedec26a2052f06d5889: Status 404 returned error can't find the container with id 91786209211b4a0aac050271ce78de6974ca604e1883cedec26a2052f06d5889 Jul 08 22:47:22.892168 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.892072 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:22.892375 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.892195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:47:22.892375 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:22.892239 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:22.892375 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:22.892239 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 22:47:22.892375 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:22.892294 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 22:47:22.892375 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:22.892370 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 22:47:22.892596 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:22.892388 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-594fb948ff-jktv6: secret "image-registry-tls" not found Jul 08 22:47:22.892596 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:22.892370 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls podName:407a5c0c-24fd-474d-b677-65c32d5c1ca5 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:23.892351929 +0000 UTC m=+38.147799628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls") pod "dns-default-xv6dw" (UID: "407a5c0c-24fd-474d-b677-65c32d5c1ca5") : secret "dns-default-metrics-tls" not found Jul 08 22:47:22.892596 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:22.892449 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert podName:2d8cc6ed-a2e9-422d-ac36-2970b983252a nodeName:}" failed. No retries permitted until 2026-07-08 22:47:23.892436357 +0000 UTC m=+38.147884042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pm98b" (UID: "2d8cc6ed-a2e9-422d-ac36-2970b983252a") : secret "networking-console-plugin-cert" not found Jul 08 22:47:22.892596 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:22.892466 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls podName:82dddf27-89f2-45f8-ba41-74a7be616df4 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:23.892455802 +0000 UTC m=+38.147903483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls") pod "image-registry-594fb948ff-jktv6" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4") : secret "image-registry-tls" not found Jul 08 22:47:23.261888 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:23.261805 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:23.262169 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:23.262010 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:23.264785 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:23.264629 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-48cc2\"" Jul 08 22:47:23.264785 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:23.264669 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 22:47:23.264785 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:23.264674 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 22:47:23.565401 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:23.565318 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jpnsr" event={"ID":"32679a71-0c9d-4b6a-8b60-130bd59ae45c","Type":"ContainerStarted","Data":"91786209211b4a0aac050271ce78de6974ca604e1883cedec26a2052f06d5889"} Jul 08 22:47:23.901600 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:23.901513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:47:23.901600 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:23.901559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:23.901833 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:23.901606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:23.901833 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:23.901683 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 22:47:23.901833 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:23.901707 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 22:47:23.901833 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:23.901746 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 22:47:23.901833 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:23.901765 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert podName:2d8cc6ed-a2e9-422d-ac36-2970b983252a nodeName:}" failed. No retries permitted until 2026-07-08 22:47:25.901745701 +0000 UTC m=+40.157193393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pm98b" (UID: "2d8cc6ed-a2e9-422d-ac36-2970b983252a") : secret "networking-console-plugin-cert" not found Jul 08 22:47:23.901833 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:23.901767 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-594fb948ff-jktv6: secret "image-registry-tls" not found Jul 08 22:47:23.901833 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:23.901785 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls podName:407a5c0c-24fd-474d-b677-65c32d5c1ca5 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:25.901775517 +0000 UTC m=+40.157223198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls") pod "dns-default-xv6dw" (UID: "407a5c0c-24fd-474d-b677-65c32d5c1ca5") : secret "dns-default-metrics-tls" not found Jul 08 22:47:23.901833 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:23.901819 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls podName:82dddf27-89f2-45f8-ba41-74a7be616df4 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:25.901801249 +0000 UTC m=+40.157248950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls") pod "image-registry-594fb948ff-jktv6" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4") : secret "image-registry-tls" not found Jul 08 22:47:25.572432 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:25.572398 2564 generic.go:356] "Generic (PLEG): container finished" podID="3168d934-7d87-498f-9733-8059ff036149" containerID="0bbbd41fc95caee47c63d16cea2b91fdaeeef96156bcab43fe084d6d97853304" exitCode=0 Jul 08 22:47:25.573141 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:25.572449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjw2b" event={"ID":"3168d934-7d87-498f-9733-8059ff036149","Type":"ContainerDied","Data":"0bbbd41fc95caee47c63d16cea2b91fdaeeef96156bcab43fe084d6d97853304"} Jul 08 22:47:25.918825 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:25.918736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:25.918969 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:25.918856 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:47:25.918969 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:25.918895 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:25.918969 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:25.918900 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 22:47:25.918969 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:25.918961 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls podName:407a5c0c-24fd-474d-b677-65c32d5c1ca5 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:29.918944361 +0000 UTC m=+44.174392041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls") pod "dns-default-xv6dw" (UID: "407a5c0c-24fd-474d-b677-65c32d5c1ca5") : secret "dns-default-metrics-tls" not found Jul 08 22:47:25.919152 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:25.919010 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 22:47:25.919152 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:25.919016 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 22:47:25.919152 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:25.919073 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert podName:2d8cc6ed-a2e9-422d-ac36-2970b983252a nodeName:}" failed. No retries permitted until 2026-07-08 22:47:29.919058537 +0000 UTC m=+44.174506223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pm98b" (UID: "2d8cc6ed-a2e9-422d-ac36-2970b983252a") : secret "networking-console-plugin-cert" not found Jul 08 22:47:25.919152 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:25.919025 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-594fb948ff-jktv6: secret "image-registry-tls" not found Jul 08 22:47:25.919279 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:25.919158 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls podName:82dddf27-89f2-45f8-ba41-74a7be616df4 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:29.919141387 +0000 UTC m=+44.174589093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls") pod "image-registry-594fb948ff-jktv6" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4") : secret "image-registry-tls" not found Jul 08 22:47:26.578928 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:26.578897 2564 generic.go:356] "Generic (PLEG): container finished" podID="3168d934-7d87-498f-9733-8059ff036149" containerID="3c9ba428de197f403705148000aa830c10b49163becb07db547d10cd3b57e946" exitCode=0 Jul 08 22:47:26.579534 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:26.578954 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjw2b" event={"ID":"3168d934-7d87-498f-9733-8059ff036149","Type":"ContainerDied","Data":"3c9ba428de197f403705148000aa830c10b49163becb07db547d10cd3b57e946"} Jul 08 22:47:26.856195 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:26.856005 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:26.956541 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:26.956514 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:27.399579 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:27.399547 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:27.448347 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:27.448265 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:27.584168 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:27.584138 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gjw2b" event={"ID":"3168d934-7d87-498f-9733-8059ff036149","Type":"ContainerStarted","Data":"0e54bf62049316499d742aceac4d1055572375d311f5e36be2b800ae1f0b9c03"} Jul 08 22:47:27.585576 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:27.585554 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jpnsr" event={"ID":"32679a71-0c9d-4b6a-8b60-130bd59ae45c","Type":"ContainerStarted","Data":"d7664c72334651438268f77b55dd61034feacdf9823712689be123d7027ec582"} Jul 08 22:47:27.604977 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:27.604936 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gjw2b" podStartSLOduration=8.107545976 podStartE2EDuration="41.6049235s" totalImagesPullingTime="33.497377524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 22:46:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:47:27.603674021 +0000 UTC m=+41.859121723" watchObservedRunningTime="2026-07-08 22:47:27.6049235 +0000 UTC m=+41.860371203" Jul 08 22:47:27.621930 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:27.621882 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jpnsr" podStartSLOduration=1.113577985 podStartE2EDuration="5.621869821s" totalImagesPullingTime="4.508291836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:47:27.621240864 +0000 UTC m=+41.876688566" watchObservedRunningTime="2026-07-08 22:47:27.621869821 +0000 UTC m=+41.877317523" Jul 08 22:47:27.734342 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:27.734257 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:27.805878 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:27.805847 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:27.936195 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:27.936148 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:28.040561 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:28.040532 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:28.136825 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:28.136793 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:28.396686 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:28.396603 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:28.454827 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:28.454789 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:29.950312 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:29.950271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:47:29.950312 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:29.950313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:29.950711 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:29.950352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:29.950711 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:29.950429 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 22:47:29.950711 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:29.950456 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 22:47:29.950711 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:29.950475 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 22:47:29.950711 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:29.950492 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-594fb948ff-jktv6: secret "image-registry-tls" not found Jul 08 22:47:29.950711 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:29.950497 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert podName:2d8cc6ed-a2e9-422d-ac36-2970b983252a nodeName:}" failed. No retries permitted until 2026-07-08 22:47:37.950480058 +0000 UTC m=+52.205927739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pm98b" (UID: "2d8cc6ed-a2e9-422d-ac36-2970b983252a") : secret "networking-console-plugin-cert" not found Jul 08 22:47:29.950711 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:29.950513 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls podName:407a5c0c-24fd-474d-b677-65c32d5c1ca5 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:37.950506781 +0000 UTC m=+52.205954461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls") pod "dns-default-xv6dw" (UID: "407a5c0c-24fd-474d-b677-65c32d5c1ca5") : secret "dns-default-metrics-tls" not found Jul 08 22:47:29.950711 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:29.950539 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls podName:82dddf27-89f2-45f8-ba41-74a7be616df4 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:37.950526455 +0000 UTC m=+52.205974136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls") pod "image-registry-594fb948ff-jktv6" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4") : secret "image-registry-tls" not found Jul 08 22:47:30.263333 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:30.263302 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:30.312486 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:30.312452 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:30.817675 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:30.817640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:31.804052 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:31.804016 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:31.857417 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:31.857383 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:32.814986 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:32.814958 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:32.889192 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:32.889155 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:32.984419 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:32.984336 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:37.833098 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:37.833038 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:38.013778 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:38.013734 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:38.013974 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:38.013836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:47:38.013974 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:38.013869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:38.013974 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:38.013879 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 22:47:38.013974 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:38.013936 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls podName:407a5c0c-24fd-474d-b677-65c32d5c1ca5 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:54.013920591 +0000 UTC m=+68.269368271 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls") pod "dns-default-xv6dw" (UID: "407a5c0c-24fd-474d-b677-65c32d5c1ca5") : secret "dns-default-metrics-tls" not found Jul 08 22:47:38.013974 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:38.013966 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 22:47:38.013974 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:38.013976 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 22:47:38.014367 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:38.013990 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-594fb948ff-jktv6: secret "image-registry-tls" not found Jul 08 22:47:38.014367 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:38.014046 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert podName:2d8cc6ed-a2e9-422d-ac36-2970b983252a nodeName:}" failed. No retries permitted until 2026-07-08 22:47:54.014029869 +0000 UTC m=+68.269477558 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pm98b" (UID: "2d8cc6ed-a2e9-422d-ac36-2970b983252a") : secret "networking-console-plugin-cert" not found Jul 08 22:47:38.014367 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:38.014061 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls podName:82dddf27-89f2-45f8-ba41-74a7be616df4 nodeName:}" failed. No retries permitted until 2026-07-08 22:47:54.014054711 +0000 UTC m=+68.269502391 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls") pod "image-registry-594fb948ff-jktv6" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4") : secret "image-registry-tls" not found Jul 08 22:47:38.908576 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:38.908519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:38.961795 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:38.961760 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:43.940239 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:43.940200 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:44.039330 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:44.039295 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:44.862299 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:44.862266 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:45.970689 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:45.970653 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:46.016424 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:46.016395 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:48.405578 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:48.405543 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:48.505890 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:48.505849 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:48.566217 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:48.566191 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pd2mh" Jul 08 22:47:48.713843 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:48.713753 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:48.766167 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:48.766127 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:48.883927 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:48.883896 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:48.933518 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:48.933468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:47:51.913280 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:51.913228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:47:51.915838 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:51.915818 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 22:47:51.923833 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:51.923816 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 22:47:51.923895 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:51.923887 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs podName:4ed6eae7-7b43-47ba-b489-136fab2a8ade nodeName:}" failed. No retries permitted until 2026-07-08 22:48:55.923866542 +0000 UTC m=+130.179314237 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs") pod "network-metrics-daemon-rldqz" (UID: "4ed6eae7-7b43-47ba-b489-136fab2a8ade") : secret "metrics-daemon-secret" not found Jul 08 22:47:52.013691 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:52.013657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:52.016550 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:52.016534 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 22:47:52.027543 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:52.027519 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0677a18f-b178-455f-a982-4c31ec52aab0-original-pull-secret\") pod \"global-pull-secret-syncer-twk7h\" (UID: \"0677a18f-b178-455f-a982-4c31ec52aab0\") " pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:52.079811 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:52.079787 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twk7h" Jul 08 22:47:52.114523 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:52.114479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz8gn\" (UniqueName: \"kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn\") pod \"network-check-target-jh8dz\" (UID: \"2a5fca6f-9523-47c5-b7f4-c6f45277ce47\") " pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:52.116992 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:52.116963 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 22:47:52.127829 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:52.127805 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 22:47:52.138755 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:52.138617 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz8gn\" (UniqueName: \"kubernetes.io/projected/2a5fca6f-9523-47c5-b7f4-c6f45277ce47-kube-api-access-sz8gn\") pod \"network-check-target-jh8dz\" (UID: \"2a5fca6f-9523-47c5-b7f4-c6f45277ce47\") " pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:52.197262 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:52.197186 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-twk7h"] Jul 08 22:47:52.200286 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:47:52.200256 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0677a18f_b178_455f_a982_4c31ec52aab0.slice/crio-daf8720e7e0b749d590314bc2e58125065eb375e1b268f9f10e7b5416726fe43 WatchSource:0}: Error finding container daf8720e7e0b749d590314bc2e58125065eb375e1b268f9f10e7b5416726fe43: Status 404 returned error can't find the container with id daf8720e7e0b749d590314bc2e58125065eb375e1b268f9f10e7b5416726fe43 Jul 08 22:47:52.274892 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:52.274868 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7tfqh\"" Jul 08 22:47:52.283012 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:52.282990 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:52.399338 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:52.399309 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jh8dz"] Jul 08 22:47:52.402357 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:47:52.402334 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5fca6f_9523_47c5_b7f4_c6f45277ce47.slice/crio-d311277b5f998c2aefa8331d40ae6826f364b815e05133a54903a1cadddc6cdf WatchSource:0}: Error finding container d311277b5f998c2aefa8331d40ae6826f364b815e05133a54903a1cadddc6cdf: Status 404 returned error can't find the container with id d311277b5f998c2aefa8331d40ae6826f364b815e05133a54903a1cadddc6cdf Jul 08 22:47:52.647641 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:52.647604 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jh8dz" event={"ID":"2a5fca6f-9523-47c5-b7f4-c6f45277ce47","Type":"ContainerStarted","Data":"d311277b5f998c2aefa8331d40ae6826f364b815e05133a54903a1cadddc6cdf"} Jul 08 22:47:52.648683 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:52.648661 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-twk7h" event={"ID":"0677a18f-b178-455f-a982-4c31ec52aab0","Type":"ContainerStarted","Data":"daf8720e7e0b749d590314bc2e58125065eb375e1b268f9f10e7b5416726fe43"} Jul 08 22:47:54.029927 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:54.029882 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:47:54.030422 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:54.029941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:47:54.030422 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:54.030000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:47:54.030422 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:54.030058 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 22:47:54.030422 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:54.030134 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 22:47:54.030422 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:54.030141 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 22:47:54.030422 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:54.030163 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-594fb948ff-jktv6: secret "image-registry-tls" not found Jul 08 22:47:54.030422 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:54.030151 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert podName:2d8cc6ed-a2e9-422d-ac36-2970b983252a nodeName:}" failed. No retries permitted until 2026-07-08 22:48:26.030129297 +0000 UTC m=+100.285576978 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pm98b" (UID: "2d8cc6ed-a2e9-422d-ac36-2970b983252a") : secret "networking-console-plugin-cert" not found Jul 08 22:47:54.030422 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:54.030229 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls podName:407a5c0c-24fd-474d-b677-65c32d5c1ca5 nodeName:}" failed. No retries permitted until 2026-07-08 22:48:26.030203942 +0000 UTC m=+100.285651623 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls") pod "dns-default-xv6dw" (UID: "407a5c0c-24fd-474d-b677-65c32d5c1ca5") : secret "dns-default-metrics-tls" not found Jul 08 22:47:54.030422 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:47:54.030246 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls podName:82dddf27-89f2-45f8-ba41-74a7be616df4 nodeName:}" failed. No retries permitted until 2026-07-08 22:48:26.030235522 +0000 UTC m=+100.285683204 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls") pod "image-registry-594fb948ff-jktv6" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4") : secret "image-registry-tls" not found Jul 08 22:47:57.663458 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:57.663419 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jh8dz" event={"ID":"2a5fca6f-9523-47c5-b7f4-c6f45277ce47","Type":"ContainerStarted","Data":"d811197f1d4df784b514612d2904318f1067573200c92604d7ecdaa141acb7ee"} Jul 08 22:47:57.663919 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:57.663524 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:47:57.664877 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:57.664851 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-twk7h" event={"ID":"0677a18f-b178-455f-a982-4c31ec52aab0","Type":"ContainerStarted","Data":"1bc9fb63497728766d74f9ac1d7340cba8a78c228f12da284f142fdd2c6ab68b"} Jul 08 22:47:57.678542 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:47:57.678499 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jh8dz" podStartSLOduration=66.982769293 podStartE2EDuration="1m11.678485319s" totalImagesPullingTime="4.695716026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:47:57.677784452 +0000 UTC m=+71.933232157" watchObservedRunningTime="2026-07-08 22:47:57.678485319 +0000 UTC m=+71.933933018" Jul 08 22:48:04.953919 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:48:04.953877 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:48:05.010222 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:48:05.010191 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:48:10.859219 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:48:10.859097 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:48:10.959822 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:48:10.959789 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:48:11.012777 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:48:11.012741 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:48:26.065943 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:48:26.065891 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:48:26.065943 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:48:26.065949 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:48:26.066494 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:48:26.065987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:48:26.066494 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:48:26.066066 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 22:48:26.066494 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:48:26.066102 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 22:48:26.066494 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:48:26.066146 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls podName:407a5c0c-24fd-474d-b677-65c32d5c1ca5 nodeName:}" failed. No retries permitted until 2026-07-08 22:49:30.066131137 +0000 UTC m=+164.321578818 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls") pod "dns-default-xv6dw" (UID: "407a5c0c-24fd-474d-b677-65c32d5c1ca5") : secret "dns-default-metrics-tls" not found Jul 08 22:48:26.066494 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:48:26.066187 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert podName:2d8cc6ed-a2e9-422d-ac36-2970b983252a nodeName:}" failed. No retries permitted until 2026-07-08 22:49:30.066168032 +0000 UTC m=+164.321615724 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pm98b" (UID: "2d8cc6ed-a2e9-422d-ac36-2970b983252a") : secret "networking-console-plugin-cert" not found Jul 08 22:48:26.066494 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:48:26.066116 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 22:48:26.066494 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:48:26.066206 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-594fb948ff-jktv6: secret "image-registry-tls" not found Jul 08 22:48:26.066494 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:48:26.066239 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls podName:82dddf27-89f2-45f8-ba41-74a7be616df4 nodeName:}" failed. No retries permitted until 2026-07-08 22:49:30.066228543 +0000 UTC m=+164.321676227 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls") pod "image-registry-594fb948ff-jktv6" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4") : secret "image-registry-tls" not found Jul 08 22:48:28.670232 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:48:28.670204 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jh8dz" Jul 08 22:48:28.686988 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:48:28.686943 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-twk7h" podStartSLOduration=96.78771307 podStartE2EDuration="1m41.686927385s" totalImagesPullingTime="4.899214315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 22:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:47:57.692063922 +0000 UTC m=+71.947511625" watchObservedRunningTime="2026-07-08 22:48:28.686927385 +0000 UTC m=+102.942375065" Jul 08 22:48:55.979706 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:48:55.979668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:48:55.980200 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:48:55.979804 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 22:48:55.980200 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:48:55.979875 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs podName:4ed6eae7-7b43-47ba-b489-136fab2a8ade nodeName:}" failed. No retries permitted until 2026-07-08 22:50:57.979859692 +0000 UTC m=+252.235307373 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs") pod "network-metrics-daemon-rldqz" (UID: "4ed6eae7-7b43-47ba-b489-136fab2a8ade") : secret "metrics-daemon-secret" not found Jul 08 22:49:25.139047 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:49:25.138840 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" podUID="82dddf27-89f2-45f8-ba41-74a7be616df4" Jul 08 22:49:25.149591 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:49:25.149549 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" podUID="2d8cc6ed-a2e9-422d-ac36-2970b983252a" Jul 08 22:49:25.179167 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:49:25.179134 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xv6dw" podUID="407a5c0c-24fd-474d-b677-65c32d5c1ca5" Jul 08 22:49:25.874287 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:49:25.874260 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:49:25.874432 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:49:25.874260 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:49:26.273652 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:49:26.273606 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:49:30.121487 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:49:30.121436 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:49:30.121487 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:49:30.121490 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:49:30.121916 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:49:30.121536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:49:30.121916 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:49:30.121587 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 22:49:30.121916 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:49:30.121639 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 22:49:30.121916 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:49:30.121644 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 22:49:30.121916 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:49:30.121661 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-594fb948ff-jktv6: secret "image-registry-tls" not found Jul 08 22:49:30.121916 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:49:30.121663 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert podName:2d8cc6ed-a2e9-422d-ac36-2970b983252a nodeName:}" failed. No retries permitted until 2026-07-08 22:51:32.121646469 +0000 UTC m=+286.377094150 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pm98b" (UID: "2d8cc6ed-a2e9-422d-ac36-2970b983252a") : secret "networking-console-plugin-cert" not found Jul 08 22:49:30.121916 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:49:30.121708 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls podName:407a5c0c-24fd-474d-b677-65c32d5c1ca5 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:32.121692677 +0000 UTC m=+286.377140360 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls") pod "dns-default-xv6dw" (UID: "407a5c0c-24fd-474d-b677-65c32d5c1ca5") : secret "dns-default-metrics-tls" not found Jul 08 22:49:30.121916 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:49:30.121725 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls podName:82dddf27-89f2-45f8-ba41-74a7be616df4 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:32.121719745 +0000 UTC m=+286.377167425 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls") pod "image-registry-594fb948ff-jktv6" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4") : secret "image-registry-tls" not found Jul 08 22:49:38.261790 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:49:38.261695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xv6dw" Jul 08 22:49:41.261368 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:49:41.261317 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:50:22.150687 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:50:22.150656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:50:37.999056 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:50:37.999012 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:50:58.030298 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:50:58.030248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:50:58.030805 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:50:58.030407 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 22:50:58.030805 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:50:58.030482 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs podName:4ed6eae7-7b43-47ba-b489-136fab2a8ade nodeName:}" failed. No retries permitted until 2026-07-08 22:53:00.030465413 +0000 UTC m=+374.285913093 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs") pod "network-metrics-daemon-rldqz" (UID: "4ed6eae7-7b43-47ba-b489-136fab2a8ade") : secret "metrics-daemon-secret" not found Jul 08 22:51:08.927506 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.927425 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm"] Jul 08 22:51:08.929459 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.929440 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:08.930997 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.930972 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp"] Jul 08 22:51:08.931806 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.931784 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 08 22:51:08.932068 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.932000 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 22:51:08.932272 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.932253 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9sj8m\"" Jul 08 22:51:08.932345 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.932326 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 22:51:08.932899 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.932883 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-xtx76"] Jul 08 22:51:08.933010 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.932996 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" Jul 08 22:51:08.934567 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.934536 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 08 22:51:08.934702 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.934687 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-sblc2"] Jul 08 22:51:08.934813 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.934796 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-xtx76" Jul 08 22:51:08.935279 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.935258 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 08 22:51:08.935437 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.935415 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 08 22:51:08.935540 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.935526 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 08 22:51:08.935606 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.935538 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 08 22:51:08.935650 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.935611 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9qhkg\"" Jul 08 22:51:08.936402 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.936385 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:08.937593 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.937576 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 08 22:51:08.937711 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.937696 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lxqgp\"" Jul 08 22:51:08.938477 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.938463 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 08 22:51:08.938729 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.938713 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pcrk4\"" Jul 08 22:51:08.938800 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.938719 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 08 22:51:08.939241 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.939226 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 08 22:51:08.939480 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.939466 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 08 22:51:08.939639 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.939626 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 08 22:51:08.944796 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.944774 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm"] Jul 08 22:51:08.947097 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.947064 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-xtx76"] Jul 08 22:51:08.952143 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.952099 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 08 22:51:08.952336 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.952312 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp"] Jul 08 22:51:08.962437 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:08.962412 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-sblc2"] Jul 08 22:51:09.004112 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.004065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpg8z\" (UniqueName: \"kubernetes.io/projected/7cb88df0-e9f8-4bec-9194-79356f674f21-kube-api-access-zpg8z\") pod \"cluster-monitoring-operator-64786d9cbb-9p9rm\" (UID: \"7cb88df0-e9f8-4bec-9194-79356f674f21\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:09.004272 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.004121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7cb88df0-e9f8-4bec-9194-79356f674f21-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-9p9rm\" (UID: \"7cb88df0-e9f8-4bec-9194-79356f674f21\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:09.004272 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.004159 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77zsk\" (UniqueName: \"kubernetes.io/projected/74c83256-c7b1-4ad4-8ba9-f8b734a2ff4d-kube-api-access-77zsk\") pod \"volume-data-source-validator-5d456fdbcd-xtx76\" (UID: \"74c83256-c7b1-4ad4-8ba9-f8b734a2ff4d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-xtx76" Jul 08 22:51:09.004272 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.004197 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7eebb5dd-85b2-45fa-8840-ab76361f826d-serving-cert\") pod \"console-operator-68b757fb67-sblc2\" (UID: \"7eebb5dd-85b2-45fa-8840-ab76361f826d\") " pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:09.004272 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.004243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e961afb-3fe2-4fad-9c78-45b9370b073d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-xgdbp\" (UID: \"1e961afb-3fe2-4fad-9c78-45b9370b073d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" Jul 08 22:51:09.004428 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.004270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-9p9rm\" (UID: \"7cb88df0-e9f8-4bec-9194-79356f674f21\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:09.004428 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.004292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nphc\" (UniqueName: \"kubernetes.io/projected/7eebb5dd-85b2-45fa-8840-ab76361f826d-kube-api-access-7nphc\") pod \"console-operator-68b757fb67-sblc2\" (UID: \"7eebb5dd-85b2-45fa-8840-ab76361f826d\") " pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:09.004428 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.004313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e961afb-3fe2-4fad-9c78-45b9370b073d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-xgdbp\" (UID: \"1e961afb-3fe2-4fad-9c78-45b9370b073d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" Jul 08 22:51:09.004428 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.004334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj7wm\" (UniqueName: \"kubernetes.io/projected/1e961afb-3fe2-4fad-9c78-45b9370b073d-kube-api-access-pj7wm\") pod \"kube-storage-version-migrator-operator-846bf668f7-xgdbp\" (UID: \"1e961afb-3fe2-4fad-9c78-45b9370b073d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" Jul 08 22:51:09.004604 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.004437 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7eebb5dd-85b2-45fa-8840-ab76361f826d-trusted-ca\") pod \"console-operator-68b757fb67-sblc2\" (UID: \"7eebb5dd-85b2-45fa-8840-ab76361f826d\") " pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:09.004604 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.004490 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7eebb5dd-85b2-45fa-8840-ab76361f826d-config\") pod \"console-operator-68b757fb67-sblc2\" (UID: \"7eebb5dd-85b2-45fa-8840-ab76361f826d\") " pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:09.021324 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.021298 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-dpvtv"] Jul 08 22:51:09.023295 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.023281 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-dpvtv" Jul 08 22:51:09.025483 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.025469 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mvfhc\"" Jul 08 22:51:09.036714 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.036694 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-dpvtv"] Jul 08 22:51:09.104956 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.104928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj7wm\" (UniqueName: \"kubernetes.io/projected/1e961afb-3fe2-4fad-9c78-45b9370b073d-kube-api-access-pj7wm\") pod \"kube-storage-version-migrator-operator-846bf668f7-xgdbp\" (UID: \"1e961afb-3fe2-4fad-9c78-45b9370b073d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" Jul 08 22:51:09.104956 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.104962 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7eebb5dd-85b2-45fa-8840-ab76361f826d-trusted-ca\") pod \"console-operator-68b757fb67-sblc2\" (UID: \"7eebb5dd-85b2-45fa-8840-ab76361f826d\") " pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:09.105206 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.104982 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slrk2\" (UniqueName: \"kubernetes.io/projected/320da31c-d68a-42f7-a6c1-90bc28e4ef35-kube-api-access-slrk2\") pod \"network-check-source-989f8b8dc-dpvtv\" (UID: \"320da31c-d68a-42f7-a6c1-90bc28e4ef35\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-dpvtv" Jul 08 22:51:09.105206 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.105001 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7eebb5dd-85b2-45fa-8840-ab76361f826d-config\") pod \"console-operator-68b757fb67-sblc2\" (UID: \"7eebb5dd-85b2-45fa-8840-ab76361f826d\") " pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:09.105206 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.105027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpg8z\" (UniqueName: \"kubernetes.io/projected/7cb88df0-e9f8-4bec-9194-79356f674f21-kube-api-access-zpg8z\") pod \"cluster-monitoring-operator-64786d9cbb-9p9rm\" (UID: \"7cb88df0-e9f8-4bec-9194-79356f674f21\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:09.105206 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.105057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7cb88df0-e9f8-4bec-9194-79356f674f21-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-9p9rm\" (UID: \"7cb88df0-e9f8-4bec-9194-79356f674f21\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:09.105206 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.105106 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77zsk\" (UniqueName: \"kubernetes.io/projected/74c83256-c7b1-4ad4-8ba9-f8b734a2ff4d-kube-api-access-77zsk\") pod \"volume-data-source-validator-5d456fdbcd-xtx76\" (UID: \"74c83256-c7b1-4ad4-8ba9-f8b734a2ff4d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-xtx76" Jul 08 22:51:09.105464 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.105281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7eebb5dd-85b2-45fa-8840-ab76361f826d-serving-cert\") pod \"console-operator-68b757fb67-sblc2\" (UID: \"7eebb5dd-85b2-45fa-8840-ab76361f826d\") " pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:09.105464 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.105335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e961afb-3fe2-4fad-9c78-45b9370b073d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-xgdbp\" (UID: \"1e961afb-3fe2-4fad-9c78-45b9370b073d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" Jul 08 22:51:09.105571 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.105493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-9p9rm\" (UID: \"7cb88df0-e9f8-4bec-9194-79356f674f21\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:09.105571 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.105540 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nphc\" (UniqueName: \"kubernetes.io/projected/7eebb5dd-85b2-45fa-8840-ab76361f826d-kube-api-access-7nphc\") pod \"console-operator-68b757fb67-sblc2\" (UID: \"7eebb5dd-85b2-45fa-8840-ab76361f826d\") " pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:09.105676 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.105582 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e961afb-3fe2-4fad-9c78-45b9370b073d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-xgdbp\" (UID: \"1e961afb-3fe2-4fad-9c78-45b9370b073d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" Jul 08 22:51:09.105676 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:09.105627 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 22:51:09.105791 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:09.105757 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls podName:7cb88df0-e9f8-4bec-9194-79356f674f21 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:09.605739661 +0000 UTC m=+263.861187342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-9p9rm" (UID: "7cb88df0-e9f8-4bec-9194-79356f674f21") : secret "cluster-monitoring-operator-tls" not found Jul 08 22:51:09.105791 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.105757 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7eebb5dd-85b2-45fa-8840-ab76361f826d-config\") pod \"console-operator-68b757fb67-sblc2\" (UID: \"7eebb5dd-85b2-45fa-8840-ab76361f826d\") " pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:09.105946 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.105922 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7cb88df0-e9f8-4bec-9194-79356f674f21-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-9p9rm\" (UID: \"7cb88df0-e9f8-4bec-9194-79356f674f21\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:09.106025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.105944 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e961afb-3fe2-4fad-9c78-45b9370b073d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-xgdbp\" (UID: \"1e961afb-3fe2-4fad-9c78-45b9370b073d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" Jul 08 22:51:09.106025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.105969 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7eebb5dd-85b2-45fa-8840-ab76361f826d-trusted-ca\") pod \"console-operator-68b757fb67-sblc2\" (UID: \"7eebb5dd-85b2-45fa-8840-ab76361f826d\") " pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:09.107835 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.107804 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e961afb-3fe2-4fad-9c78-45b9370b073d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-xgdbp\" (UID: \"1e961afb-3fe2-4fad-9c78-45b9370b073d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" Jul 08 22:51:09.107934 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.107878 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7eebb5dd-85b2-45fa-8840-ab76361f826d-serving-cert\") pod \"console-operator-68b757fb67-sblc2\" (UID: \"7eebb5dd-85b2-45fa-8840-ab76361f826d\") " pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:09.112607 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.112578 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj7wm\" (UniqueName: \"kubernetes.io/projected/1e961afb-3fe2-4fad-9c78-45b9370b073d-kube-api-access-pj7wm\") pod \"kube-storage-version-migrator-operator-846bf668f7-xgdbp\" (UID: \"1e961afb-3fe2-4fad-9c78-45b9370b073d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" Jul 08 22:51:09.112740 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.112708 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77zsk\" (UniqueName: \"kubernetes.io/projected/74c83256-c7b1-4ad4-8ba9-f8b734a2ff4d-kube-api-access-77zsk\") pod \"volume-data-source-validator-5d456fdbcd-xtx76\" (UID: \"74c83256-c7b1-4ad4-8ba9-f8b734a2ff4d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-xtx76" Jul 08 22:51:09.112895 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.112868 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpg8z\" (UniqueName: \"kubernetes.io/projected/7cb88df0-e9f8-4bec-9194-79356f674f21-kube-api-access-zpg8z\") pod \"cluster-monitoring-operator-64786d9cbb-9p9rm\" (UID: \"7cb88df0-e9f8-4bec-9194-79356f674f21\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:09.114220 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.114203 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nphc\" (UniqueName: \"kubernetes.io/projected/7eebb5dd-85b2-45fa-8840-ab76361f826d-kube-api-access-7nphc\") pod \"console-operator-68b757fb67-sblc2\" (UID: \"7eebb5dd-85b2-45fa-8840-ab76361f826d\") " pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:09.182962 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.182885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:09.206558 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.206532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slrk2\" (UniqueName: \"kubernetes.io/projected/320da31c-d68a-42f7-a6c1-90bc28e4ef35-kube-api-access-slrk2\") pod \"network-check-source-989f8b8dc-dpvtv\" (UID: \"320da31c-d68a-42f7-a6c1-90bc28e4ef35\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-dpvtv" Jul 08 22:51:09.213688 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.213656 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slrk2\" (UniqueName: \"kubernetes.io/projected/320da31c-d68a-42f7-a6c1-90bc28e4ef35-kube-api-access-slrk2\") pod \"network-check-source-989f8b8dc-dpvtv\" (UID: \"320da31c-d68a-42f7-a6c1-90bc28e4ef35\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-dpvtv" Jul 08 22:51:09.249263 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.249238 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" Jul 08 22:51:09.257844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.257694 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-xtx76" Jul 08 22:51:09.262116 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.262033 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:09.284589 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.284558 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:09.332116 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.331468 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-dpvtv" Jul 08 22:51:09.402037 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.401285 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp"] Jul 08 22:51:09.473760 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.473691 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-dpvtv"] Jul 08 22:51:09.476551 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:09.476527 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320da31c_d68a_42f7_a6c1_90bc28e4ef35.slice/crio-079b1e186ce040ff54249da598c257de0650f1c8918e89bccc9cdbb5e852e24f WatchSource:0}: Error finding container 079b1e186ce040ff54249da598c257de0650f1c8918e89bccc9cdbb5e852e24f: Status 404 returned error can't find the container with id 079b1e186ce040ff54249da598c257de0650f1c8918e89bccc9cdbb5e852e24f Jul 08 22:51:09.610412 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.610380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-9p9rm\" (UID: \"7cb88df0-e9f8-4bec-9194-79356f674f21\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:09.610579 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:09.610553 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 22:51:09.610636 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:09.610625 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls podName:7cb88df0-e9f8-4bec-9194-79356f674f21 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:10.610606774 +0000 UTC m=+264.866054455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-9p9rm" (UID: "7cb88df0-e9f8-4bec-9194-79356f674f21") : secret "cluster-monitoring-operator-tls" not found Jul 08 22:51:09.621261 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.621233 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-xtx76"] Jul 08 22:51:09.624102 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:09.624060 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c83256_c7b1_4ad4_8ba9_f8b734a2ff4d.slice/crio-bc6690e802e4f969d4127233c25350a4bc23b27a16875b086d110db8186d7354 WatchSource:0}: Error finding container bc6690e802e4f969d4127233c25350a4bc23b27a16875b086d110db8186d7354: Status 404 returned error can't find the container with id bc6690e802e4f969d4127233c25350a4bc23b27a16875b086d110db8186d7354 Jul 08 22:51:09.624204 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:09.624136 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-sblc2"] Jul 08 22:51:09.627245 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:09.627224 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eebb5dd_85b2_45fa_8840_ab76361f826d.slice/crio-0bd05166771a4c7a9776a4c3b23af868f2749b40fdbb1c5c32c4151bf1c58097 WatchSource:0}: Error finding container 0bd05166771a4c7a9776a4c3b23af868f2749b40fdbb1c5c32c4151bf1c58097: Status 404 returned error can't find the container with id 0bd05166771a4c7a9776a4c3b23af868f2749b40fdbb1c5c32c4151bf1c58097 Jul 08 22:51:10.119997 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:10.119957 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-xtx76" event={"ID":"74c83256-c7b1-4ad4-8ba9-f8b734a2ff4d","Type":"ContainerStarted","Data":"bc6690e802e4f969d4127233c25350a4bc23b27a16875b086d110db8186d7354"} Jul 08 22:51:10.121341 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:10.121287 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" event={"ID":"1e961afb-3fe2-4fad-9c78-45b9370b073d","Type":"ContainerStarted","Data":"23e2d1652709bf3bc0ea1957e910c33a9066a3b20d3f894ea89637cea63bb2ff"} Jul 08 22:51:10.123066 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:10.123022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-dpvtv" event={"ID":"320da31c-d68a-42f7-a6c1-90bc28e4ef35","Type":"ContainerStarted","Data":"4115594223ec7ca39cd0f1362822759ac13a64f2e772a8c92cb81df45928c487"} Jul 08 22:51:10.123237 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:10.123073 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-dpvtv" event={"ID":"320da31c-d68a-42f7-a6c1-90bc28e4ef35","Type":"ContainerStarted","Data":"079b1e186ce040ff54249da598c257de0650f1c8918e89bccc9cdbb5e852e24f"} Jul 08 22:51:10.124807 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:10.124768 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-sblc2" event={"ID":"7eebb5dd-85b2-45fa-8840-ab76361f826d","Type":"ContainerStarted","Data":"0bd05166771a4c7a9776a4c3b23af868f2749b40fdbb1c5c32c4151bf1c58097"} Jul 08 22:51:10.136839 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:10.136767 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-dpvtv" podStartSLOduration=1.136753612 podStartE2EDuration="1.136753612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:10.136737029 +0000 UTC m=+264.392184733" watchObservedRunningTime="2026-07-08 22:51:10.136753612 +0000 UTC m=+264.392201314" Jul 08 22:51:10.616552 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:10.616511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-9p9rm\" (UID: \"7cb88df0-e9f8-4bec-9194-79356f674f21\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:10.616759 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:10.616740 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 22:51:10.616836 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:10.616814 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls podName:7cb88df0-e9f8-4bec-9194-79356f674f21 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:12.616793491 +0000 UTC m=+266.872241185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-9p9rm" (UID: "7cb88df0-e9f8-4bec-9194-79356f674f21") : secret "cluster-monitoring-operator-tls" not found Jul 08 22:51:12.132638 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.132598 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-xtx76" event={"ID":"74c83256-c7b1-4ad4-8ba9-f8b734a2ff4d","Type":"ContainerStarted","Data":"a1f5cc694f43010d77b44eea5d908181e89449f18d6b827bb1ae26ac5e6aa5af"} Jul 08 22:51:12.133992 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.133962 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" event={"ID":"1e961afb-3fe2-4fad-9c78-45b9370b073d","Type":"ContainerStarted","Data":"1ca96a6fe34604479f862c10b5ab0c239072c9fd5303d486cf469f2450e6831c"} Jul 08 22:51:12.146775 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.146733 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-xtx76" podStartSLOduration=2.248028006 podStartE2EDuration="4.146716381s" totalImagesPullingTime="1.898688375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:12.146092961 +0000 UTC m=+266.401540657" watchObservedRunningTime="2026-07-08 22:51:12.146716381 +0000 UTC m=+266.402164089" Jul 08 22:51:12.163534 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.163484 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" podStartSLOduration=2.0224652 podStartE2EDuration="4.163464392s" totalImagesPullingTime="2.140999192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:12.162564527 +0000 UTC m=+266.418012231" watchObservedRunningTime="2026-07-08 22:51:12.163464392 +0000 UTC m=+266.418912097" Jul 08 22:51:12.268366 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.268340 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:12.368348 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.368309 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:12.582137 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.582103 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-77fdr"] Jul 08 22:51:12.584268 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.584251 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:12.587284 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.587258 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 22:51:12.587811 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.587796 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 22:51:12.587918 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.587835 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 22:51:12.588169 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.588150 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 22:51:12.588553 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.588537 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gcg7c\"" Jul 08 22:51:12.599403 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.599375 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-77fdr"] Jul 08 22:51:12.616556 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.616499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:12.635621 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.635594 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:12.635717 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.635625 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2mm\" (UniqueName: \"kubernetes.io/projected/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-api-access-xb2mm\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:12.635717 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.635647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:12.635717 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.635697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1cc7e87-c580-4f51-8870-9ed1cbe58166-data-volume\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:12.635828 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.635799 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1cc7e87-c580-4f51-8870-9ed1cbe58166-crio-socket\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:12.635871 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.635839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-9p9rm\" (UID: \"7cb88df0-e9f8-4bec-9194-79356f674f21\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:12.635952 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:12.635939 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 22:51:12.636002 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:12.635993 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls podName:7cb88df0-e9f8-4bec-9194-79356f674f21 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:16.635981149 +0000 UTC m=+270.891428833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-9p9rm" (UID: "7cb88df0-e9f8-4bec-9194-79356f674f21") : secret "cluster-monitoring-operator-tls" not found Jul 08 22:51:12.736527 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.736501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1cc7e87-c580-4f51-8870-9ed1cbe58166-crio-socket\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:12.736671 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.736547 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:12.736671 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.736564 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2mm\" (UniqueName: \"kubernetes.io/projected/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-api-access-xb2mm\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:12.736671 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.736623 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:12.736671 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.736637 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1cc7e87-c580-4f51-8870-9ed1cbe58166-crio-socket\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:12.736671 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.736655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1cc7e87-c580-4f51-8870-9ed1cbe58166-data-volume\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:12.736853 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:12.736642 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 22:51:12.736853 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:12.736765 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm podName:e1cc7e87-c580-4f51-8870-9ed1cbe58166 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:13.236751162 +0000 UTC m=+267.492198844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm") pod "insights-runtime-extractor-77fdr" (UID: "e1cc7e87-c580-4f51-8870-9ed1cbe58166") : configmap "kube-rbac-proxy" not found Jul 08 22:51:12.736853 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:12.736706 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 22:51:12.736964 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:12.736865 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls podName:e1cc7e87-c580-4f51-8870-9ed1cbe58166 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:13.236845485 +0000 UTC m=+267.492293168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls") pod "insights-runtime-extractor-77fdr" (UID: "e1cc7e87-c580-4f51-8870-9ed1cbe58166") : secret "insights-runtime-extractor-tls" not found Jul 08 22:51:12.736964 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.736957 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1cc7e87-c580-4f51-8870-9ed1cbe58166-data-volume\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:12.745002 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:12.744979 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2mm\" (UniqueName: \"kubernetes.io/projected/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-api-access-xb2mm\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:13.103797 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.103760 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:13.131448 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.131416 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-67rtd"] Jul 08 22:51:13.133577 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.133560 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-67rtd" Jul 08 22:51:13.136150 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.136124 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 08 22:51:13.136277 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.136125 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 08 22:51:13.136277 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.136162 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hwtqh\"" Jul 08 22:51:13.140298 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.140277 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/0.log" Jul 08 22:51:13.140411 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.140312 2564 generic.go:356] "Generic (PLEG): container finished" podID="7eebb5dd-85b2-45fa-8840-ab76361f826d" containerID="acb9143399d5a4d3911db8e115cb57f8ffa8e1f23f038f337427399f869a8932" exitCode=255 Jul 08 22:51:13.140411 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.140363 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-sblc2" event={"ID":"7eebb5dd-85b2-45fa-8840-ab76361f826d","Type":"ContainerDied","Data":"acb9143399d5a4d3911db8e115cb57f8ffa8e1f23f038f337427399f869a8932"} Jul 08 22:51:13.140667 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.140648 2564 scope.go:117] "RemoveContainer" containerID="acb9143399d5a4d3911db8e115cb57f8ffa8e1f23f038f337427399f869a8932" Jul 08 22:51:13.143691 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.143667 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-67rtd"] Jul 08 22:51:13.164167 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.164131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:13.227370 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.227342 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:13.241702 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.241620 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmwkz\" (UniqueName: \"kubernetes.io/projected/35a16af7-2907-4ca1-a490-c0450ef37cbe-kube-api-access-jmwkz\") pod \"migrator-d57d58cc9-67rtd\" (UID: \"35a16af7-2907-4ca1-a490-c0450ef37cbe\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-67rtd" Jul 08 22:51:13.242035 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.241780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:13.242035 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.241837 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:13.242204 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:13.242157 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 22:51:13.242261 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:13.242218 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 22:51:13.242261 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:13.242229 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls podName:e1cc7e87-c580-4f51-8870-9ed1cbe58166 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:14.242207594 +0000 UTC m=+268.497655275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls") pod "insights-runtime-extractor-77fdr" (UID: "e1cc7e87-c580-4f51-8870-9ed1cbe58166") : secret "insights-runtime-extractor-tls" not found Jul 08 22:51:13.242405 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:13.242279 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm podName:e1cc7e87-c580-4f51-8870-9ed1cbe58166 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:14.242249151 +0000 UTC m=+268.497696832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm") pod "insights-runtime-extractor-77fdr" (UID: "e1cc7e87-c580-4f51-8870-9ed1cbe58166") : configmap "kube-rbac-proxy" not found Jul 08 22:51:13.328510 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.328476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:13.342725 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.342695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmwkz\" (UniqueName: \"kubernetes.io/projected/35a16af7-2907-4ca1-a490-c0450ef37cbe-kube-api-access-jmwkz\") pod \"migrator-d57d58cc9-67rtd\" (UID: \"35a16af7-2907-4ca1-a490-c0450ef37cbe\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-67rtd" Jul 08 22:51:13.352352 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.352328 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmwkz\" (UniqueName: \"kubernetes.io/projected/35a16af7-2907-4ca1-a490-c0450ef37cbe-kube-api-access-jmwkz\") pod \"migrator-d57d58cc9-67rtd\" (UID: \"35a16af7-2907-4ca1-a490-c0450ef37cbe\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-67rtd" Jul 08 22:51:13.444231 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.444144 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-67rtd" Jul 08 22:51:13.494990 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.494958 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:13.542514 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.542476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:13.566150 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.566027 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-67rtd"] Jul 08 22:51:13.568429 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:13.568401 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a16af7_2907_4ca1_a490_c0450ef37cbe.slice/crio-ea2c32368861edc41e2553b1e5519c5359721cbb4089cd0d3867d19b32e325af WatchSource:0}: Error finding container ea2c32368861edc41e2553b1e5519c5359721cbb4089cd0d3867d19b32e325af: Status 404 returned error can't find the container with id ea2c32368861edc41e2553b1e5519c5359721cbb4089cd0d3867d19b32e325af Jul 08 22:51:13.972495 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:13.972454 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:14.076146 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:14.076115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:14.144711 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:14.144685 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 22:51:14.145124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:14.145034 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/0.log" Jul 08 22:51:14.145124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:14.145069 2564 generic.go:356] "Generic (PLEG): container finished" podID="7eebb5dd-85b2-45fa-8840-ab76361f826d" containerID="25ba78e14a2dcd46e8a772252f484d9aca8688a2dad71d6a9257dfb479e3858a" exitCode=255 Jul 08 22:51:14.145210 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:14.145154 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-sblc2" event={"ID":"7eebb5dd-85b2-45fa-8840-ab76361f826d","Type":"ContainerDied","Data":"25ba78e14a2dcd46e8a772252f484d9aca8688a2dad71d6a9257dfb479e3858a"} Jul 08 22:51:14.145255 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:14.145236 2564 scope.go:117] "RemoveContainer" containerID="acb9143399d5a4d3911db8e115cb57f8ffa8e1f23f038f337427399f869a8932" Jul 08 22:51:14.145484 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:14.145458 2564 scope.go:117] "RemoveContainer" containerID="25ba78e14a2dcd46e8a772252f484d9aca8688a2dad71d6a9257dfb479e3858a" Jul 08 22:51:14.145715 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:14.145692 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-sblc2_openshift-console-operator(7eebb5dd-85b2-45fa-8840-ab76361f826d)\"" pod="openshift-console-operator/console-operator-68b757fb67-sblc2" podUID="7eebb5dd-85b2-45fa-8840-ab76361f826d" Jul 08 22:51:14.146496 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:14.146363 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-67rtd" event={"ID":"35a16af7-2907-4ca1-a490-c0450ef37cbe","Type":"ContainerStarted","Data":"ea2c32368861edc41e2553b1e5519c5359721cbb4089cd0d3867d19b32e325af"} Jul 08 22:51:14.249671 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:14.249640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:14.249852 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:14.249685 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:14.249852 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:14.249742 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 22:51:14.249852 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:14.249814 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm podName:e1cc7e87-c580-4f51-8870-9ed1cbe58166 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:16.249792626 +0000 UTC m=+270.505240314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm") pod "insights-runtime-extractor-77fdr" (UID: "e1cc7e87-c580-4f51-8870-9ed1cbe58166") : configmap "kube-rbac-proxy" not found Jul 08 22:51:14.250025 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:14.249851 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 22:51:14.250025 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:14.249926 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls podName:e1cc7e87-c580-4f51-8870-9ed1cbe58166 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:16.249905717 +0000 UTC m=+270.505353398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls") pod "insights-runtime-extractor-77fdr" (UID: "e1cc7e87-c580-4f51-8870-9ed1cbe58166") : secret "insights-runtime-extractor-tls" not found Jul 08 22:51:14.377529 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:14.377503 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-297fk_eaf82025-12e7-4d8d-a76a-127dd040d050/dns-node-resolver/0.log" Jul 08 22:51:15.151666 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:15.151628 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-67rtd" event={"ID":"35a16af7-2907-4ca1-a490-c0450ef37cbe","Type":"ContainerStarted","Data":"cb1313a1f467d4b964c3cf7aca125dbf2826eeef4df7e7f224d6c53fba7b24b9"} Jul 08 22:51:15.152095 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:15.151695 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-67rtd" event={"ID":"35a16af7-2907-4ca1-a490-c0450ef37cbe","Type":"ContainerStarted","Data":"b38314147cee37a03483fdc1c61b17a37db509d64e7e722714997b1469e28f11"} Jul 08 22:51:15.153036 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:15.153018 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 22:51:15.153318 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:15.153304 2564 scope.go:117] "RemoveContainer" containerID="25ba78e14a2dcd46e8a772252f484d9aca8688a2dad71d6a9257dfb479e3858a" Jul 08 22:51:15.153454 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:15.153437 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-sblc2_openshift-console-operator(7eebb5dd-85b2-45fa-8840-ab76361f826d)\"" pod="openshift-console-operator/console-operator-68b757fb67-sblc2" podUID="7eebb5dd-85b2-45fa-8840-ab76361f826d" Jul 08 22:51:15.166149 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:15.166102 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-67rtd" podStartSLOduration=0.8095876 podStartE2EDuration="2.166068447s" totalImagesPullingTime="1.356480847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:15.16512697 +0000 UTC m=+269.420574679" watchObservedRunningTime="2026-07-08 22:51:15.166068447 +0000 UTC m=+269.421516173" Jul 08 22:51:15.230977 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:15.230907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:15.276282 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:15.276252 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:15.581397 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:15.581372 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h7dmt_9c0a262a-d0f9-4fd3-91dc-44848d6d50b2/node-ca/0.log" Jul 08 22:51:15.672818 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:15.672779 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:15.771958 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:15.771926 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:16.178327 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:16.178297 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jpnsr_32679a71-0c9d-4b6a-8b60-130bd59ae45c/serve-healthcheck-canary/0.log" Jul 08 22:51:16.265504 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:16.265475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:16.265504 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:16.265508 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:16.265697 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:16.265587 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 22:51:16.265697 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:16.265609 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 22:51:16.265697 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:16.265632 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls podName:e1cc7e87-c580-4f51-8870-9ed1cbe58166 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:20.265620033 +0000 UTC m=+274.521067714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls") pod "insights-runtime-extractor-77fdr" (UID: "e1cc7e87-c580-4f51-8870-9ed1cbe58166") : secret "insights-runtime-extractor-tls" not found Jul 08 22:51:16.265697 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:16.265658 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm podName:e1cc7e87-c580-4f51-8870-9ed1cbe58166 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:20.265646176 +0000 UTC m=+274.521093857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm") pod "insights-runtime-extractor-77fdr" (UID: "e1cc7e87-c580-4f51-8870-9ed1cbe58166") : configmap "kube-rbac-proxy" not found Jul 08 22:51:16.417731 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:16.417701 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:16.668155 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:16.668117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-9p9rm\" (UID: \"7cb88df0-e9f8-4bec-9194-79356f674f21\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:16.668329 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:16.668260 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 22:51:16.668329 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:16.668319 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls podName:7cb88df0-e9f8-4bec-9194-79356f674f21 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:24.6683037 +0000 UTC m=+278.923751381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-9p9rm" (UID: "7cb88df0-e9f8-4bec-9194-79356f674f21") : secret "cluster-monitoring-operator-tls" not found Jul 08 22:51:17.367194 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:17.367158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:17.516490 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:17.516450 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:17.568173 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:17.568142 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:17.719147 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:17.719049 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:17.783072 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:17.783038 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:18.070269 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:18.070226 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:19.070362 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:19.070334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:19.168401 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:19.168370 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:19.262191 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:19.262159 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:19.262344 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:19.262215 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:19.262599 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:19.262584 2564 scope.go:117] "RemoveContainer" containerID="25ba78e14a2dcd46e8a772252f484d9aca8688a2dad71d6a9257dfb479e3858a" Jul 08 22:51:19.262777 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:19.262757 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-sblc2_openshift-console-operator(7eebb5dd-85b2-45fa-8840-ab76361f826d)\"" pod="openshift-console-operator/console-operator-68b757fb67-sblc2" podUID="7eebb5dd-85b2-45fa-8840-ab76361f826d" Jul 08 22:51:19.472524 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:19.472438 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:19.663154 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:19.663118 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:19.860456 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:19.860423 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:20.261823 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:20.261791 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:20.300764 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:20.300725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:20.300764 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:20.300764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:20.300974 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:20.300844 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 22:51:20.300974 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:20.300886 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 22:51:20.300974 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:20.300911 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm podName:e1cc7e87-c580-4f51-8870-9ed1cbe58166 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:28.300896444 +0000 UTC m=+282.556344125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm") pod "insights-runtime-extractor-77fdr" (UID: "e1cc7e87-c580-4f51-8870-9ed1cbe58166") : configmap "kube-rbac-proxy" not found Jul 08 22:51:20.300974 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:20.300938 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls podName:e1cc7e87-c580-4f51-8870-9ed1cbe58166 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:28.300925752 +0000 UTC m=+282.556373433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls") pod "insights-runtime-extractor-77fdr" (UID: "e1cc7e87-c580-4f51-8870-9ed1cbe58166") : secret "insights-runtime-extractor-tls" not found Jul 08 22:51:21.066172 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:21.066134 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:21.162098 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:21.162050 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:21.262525 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:21.262491 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:21.363166 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:21.363066 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:21.661754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:21.661665 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:21.858448 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:21.858418 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:22.063279 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:22.063251 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:22.463898 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:22.463806 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:22.864105 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:22.864048 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:23.062351 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:23.062320 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:23.459311 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:23.459279 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:23.877004 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:23.876973 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:24.061507 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:24.061477 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:24.461122 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:24.461065 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:24.734393 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:24.734307 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-9p9rm\" (UID: \"7cb88df0-e9f8-4bec-9194-79356f674f21\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:24.736703 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:24.736683 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cb88df0-e9f8-4bec-9194-79356f674f21-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-9p9rm\" (UID: \"7cb88df0-e9f8-4bec-9194-79356f674f21\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:24.839688 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:24.839657 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" Jul 08 22:51:24.861787 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:24.861757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:24.957010 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:24.956980 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm"] Jul 08 22:51:24.959761 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:24.959723 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb88df0_e9f8_4bec_9194_79356f674f21.slice/crio-6b64ace13234300d7e8573fd7fa531177e583381eb6ca8ad6c543b59853c2ad3 WatchSource:0}: Error finding container 6b64ace13234300d7e8573fd7fa531177e583381eb6ca8ad6c543b59853c2ad3: Status 404 returned error can't find the container with id 6b64ace13234300d7e8573fd7fa531177e583381eb6ca8ad6c543b59853c2ad3 Jul 08 22:51:25.183375 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:25.183343 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" event={"ID":"7cb88df0-e9f8-4bec-9194-79356f674f21","Type":"ContainerStarted","Data":"6b64ace13234300d7e8573fd7fa531177e583381eb6ca8ad6c543b59853c2ad3"} Jul 08 22:51:25.257450 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:25.257423 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:25.663643 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:25.663593 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:25.854990 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:25.854911 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:25.913286 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:25.913253 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:26.259629 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:26.259593 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:27.048558 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:27.048477 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:27.148447 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:27.148412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:27.190893 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:27.190865 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" event={"ID":"7cb88df0-e9f8-4bec-9194-79356f674f21","Type":"ContainerStarted","Data":"e4309b1fde842b85e4d8574702e81a22981544cdc670c85a0afe74e2c598cf2d"} Jul 08 22:51:27.206424 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:27.206378 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-9p9rm" podStartSLOduration=17.417379156 podStartE2EDuration="19.206364761s" totalImagesPullingTime="1.788985605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:27.204972574 +0000 UTC m=+281.460420276" watchObservedRunningTime="2026-07-08 22:51:27.206364761 +0000 UTC m=+281.461812463" Jul 08 22:51:27.208917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:27.208892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:27.464239 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:27.464151 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:27.664345 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:27.664309 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:27.862608 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:27.862578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:28.062695 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:28.062665 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:28.362785 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:28.362755 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:28.362977 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:28.362794 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:28.363325 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:28.363300 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1cc7e87-c580-4f51-8870-9ed1cbe58166-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:28.365155 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:28.365139 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1cc7e87-c580-4f51-8870-9ed1cbe58166-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77fdr\" (UID: \"e1cc7e87-c580-4f51-8870-9ed1cbe58166\") " pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:28.464352 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:28.463906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:28.493612 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:28.493589 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-77fdr" Jul 08 22:51:28.617636 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:28.617559 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-77fdr"] Jul 08 22:51:28.620579 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:28.620547 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1cc7e87_c580_4f51_8870_9ed1cbe58166.slice/crio-9b47075bbf94eab7525d82620b220c2fdc0ba5a5dcb1a249dae6bf92a8c81e37 WatchSource:0}: Error finding container 9b47075bbf94eab7525d82620b220c2fdc0ba5a5dcb1a249dae6bf92a8c81e37: Status 404 returned error can't find the container with id 9b47075bbf94eab7525d82620b220c2fdc0ba5a5dcb1a249dae6bf92a8c81e37 Jul 08 22:51:28.875435 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:28.875350 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" podUID="2d8cc6ed-a2e9-422d-ac36-2970b983252a" Jul 08 22:51:28.875435 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:28.875363 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" podUID="82dddf27-89f2-45f8-ba41-74a7be616df4" Jul 08 22:51:29.202548 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:29.202475 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:51:29.203188 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:29.203162 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77fdr" event={"ID":"e1cc7e87-c580-4f51-8870-9ed1cbe58166","Type":"ContainerStarted","Data":"9b47075bbf94eab7525d82620b220c2fdc0ba5a5dcb1a249dae6bf92a8c81e37"} Jul 08 22:51:29.203402 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:29.203235 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:51:30.262683 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:30.262615 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:30.313503 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:30.313466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:32.196077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.196036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:51:32.196557 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.196119 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:51:32.196557 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.196149 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:51:32.198925 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.198899 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/407a5c0c-24fd-474d-b677-65c32d5c1ca5-metrics-tls\") pod \"dns-default-xv6dw\" (UID: \"407a5c0c-24fd-474d-b677-65c32d5c1ca5\") " pod="openshift-dns/dns-default-xv6dw" Jul 08 22:51:32.199053 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.198965 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls\") pod \"image-registry-594fb948ff-jktv6\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:51:32.199053 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.199033 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d8cc6ed-a2e9-422d-ac36-2970b983252a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pm98b\" (UID: \"2d8cc6ed-a2e9-422d-ac36-2970b983252a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:51:32.207529 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.207507 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-r8m6j\"" Jul 08 22:51:32.207529 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.207520 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cvqzh\"" Jul 08 22:51:32.214098 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.214069 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" Jul 08 22:51:32.214209 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.214117 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:51:32.265155 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.265037 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fbpb9\"" Jul 08 22:51:32.273932 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.272674 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xv6dw" Jul 08 22:51:32.362790 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.362367 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-pm98b"] Jul 08 22:51:32.366935 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:32.366014 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8cc6ed_a2e9_422d_ac36_2970b983252a.slice/crio-19536afea2340e68c97096abcdf7e9265195295582b4025b41571765aadee39e WatchSource:0}: Error finding container 19536afea2340e68c97096abcdf7e9265195295582b4025b41571765aadee39e: Status 404 returned error can't find the container with id 19536afea2340e68c97096abcdf7e9265195295582b4025b41571765aadee39e Jul 08 22:51:32.389582 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.389554 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-594fb948ff-jktv6"] Jul 08 22:51:32.393025 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:32.392991 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82dddf27_89f2_45f8_ba41_74a7be616df4.slice/crio-5e74f25f0a303986cdebcab341c6e88ccf8163b9a42a194f341bafcbb5ad9172 WatchSource:0}: Error finding container 5e74f25f0a303986cdebcab341c6e88ccf8163b9a42a194f341bafcbb5ad9172: Status 404 returned error can't find the container with id 5e74f25f0a303986cdebcab341c6e88ccf8163b9a42a194f341bafcbb5ad9172 Jul 08 22:51:32.430327 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:32.430297 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xv6dw"] Jul 08 22:51:32.433498 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:32.433471 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407a5c0c_24fd_474d_b677_65c32d5c1ca5.slice/crio-8ac2246bd0694120b7f374f5d9fca7c3515037def11c755c954d043f2ae76e97 WatchSource:0}: Error finding container 8ac2246bd0694120b7f374f5d9fca7c3515037def11c755c954d043f2ae76e97: Status 404 returned error can't find the container with id 8ac2246bd0694120b7f374f5d9fca7c3515037def11c755c954d043f2ae76e97 Jul 08 22:51:33.217737 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:33.217688 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" event={"ID":"82dddf27-89f2-45f8-ba41-74a7be616df4","Type":"ContainerStarted","Data":"1b92a9584be49ed84db4ee3134d4ac8c6b4e9575a3f571d7c1745380390014a0"} Jul 08 22:51:33.218184 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:33.217742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" event={"ID":"82dddf27-89f2-45f8-ba41-74a7be616df4","Type":"ContainerStarted","Data":"5e74f25f0a303986cdebcab341c6e88ccf8163b9a42a194f341bafcbb5ad9172"} Jul 08 22:51:33.218184 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:33.217769 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:51:33.219722 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:33.219699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77fdr" event={"ID":"e1cc7e87-c580-4f51-8870-9ed1cbe58166","Type":"ContainerStarted","Data":"e283c96c9428a7e88fc0570ef7b2d702c46cd8fb3fc3d56b8b5ac6475ff65853"} Jul 08 22:51:33.219844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:33.219743 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77fdr" event={"ID":"e1cc7e87-c580-4f51-8870-9ed1cbe58166","Type":"ContainerStarted","Data":"e069d0dc058dd5cabc66d2f546ae5b3989fabf7c6c894d8d0e6113871094499c"} Jul 08 22:51:33.221417 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:33.221393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" event={"ID":"2d8cc6ed-a2e9-422d-ac36-2970b983252a","Type":"ContainerStarted","Data":"19536afea2340e68c97096abcdf7e9265195295582b4025b41571765aadee39e"} Jul 08 22:51:33.222557 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:33.222533 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xv6dw" event={"ID":"407a5c0c-24fd-474d-b677-65c32d5c1ca5","Type":"ContainerStarted","Data":"8ac2246bd0694120b7f374f5d9fca7c3515037def11c755c954d043f2ae76e97"} Jul 08 22:51:33.238289 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:33.238184 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" podStartSLOduration=263.238165321 podStartE2EDuration="4m23.238165321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:33.236316548 +0000 UTC m=+287.491764252" watchObservedRunningTime="2026-07-08 22:51:33.238165321 +0000 UTC m=+287.493613028" Jul 08 22:51:33.262439 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:33.262380 2564 scope.go:117] "RemoveContainer" containerID="25ba78e14a2dcd46e8a772252f484d9aca8688a2dad71d6a9257dfb479e3858a" Jul 08 22:51:34.202358 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.201967 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j9692"] Jul 08 22:51:34.205398 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.205339 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j9692" Jul 08 22:51:34.209235 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.209210 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 08 22:51:34.209380 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.209308 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-r8s2h\"" Jul 08 22:51:34.219048 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.218991 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j9692"] Jul 08 22:51:34.228819 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.228788 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" event={"ID":"2d8cc6ed-a2e9-422d-ac36-2970b983252a","Type":"ContainerStarted","Data":"ea0a161b47e8ff6a49f669fc537d13be84cbe86f5e5581ced412be7cc2c83c14"} Jul 08 22:51:34.231047 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.231020 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 22:51:34.231152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.231129 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-sblc2" event={"ID":"7eebb5dd-85b2-45fa-8840-ab76361f826d","Type":"ContainerStarted","Data":"3b79af78c3f323a45eb0048cb0ad9903b0e5f38669e5778339b8583637b04a92"} Jul 08 22:51:34.231770 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.231604 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:34.254345 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.251676 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pm98b" podStartSLOduration=262.063189341 podStartE2EDuration="4m23.251658379s" totalImagesPullingTime="1.188469038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:34.249285824 +0000 UTC m=+288.504733527" watchObservedRunningTime="2026-07-08 22:51:34.251658379 +0000 UTC m=+288.507106083" Jul 08 22:51:34.310705 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.310666 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:34.314433 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.313993 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4f41cd53-0ea2-49fa-8677-683e3abf8150-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-j9692\" (UID: \"4f41cd53-0ea2-49fa-8677-683e3abf8150\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j9692" Jul 08 22:51:34.409201 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.409141 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:34.416179 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.415305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4f41cd53-0ea2-49fa-8677-683e3abf8150-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-j9692\" (UID: \"4f41cd53-0ea2-49fa-8677-683e3abf8150\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j9692" Jul 08 22:51:34.419750 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.419714 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4f41cd53-0ea2-49fa-8677-683e3abf8150-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-j9692\" (UID: \"4f41cd53-0ea2-49fa-8677-683e3abf8150\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j9692" Jul 08 22:51:34.509545 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.509506 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:34.520550 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.520515 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j9692" Jul 08 22:51:34.565322 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.565291 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-sblc2" Jul 08 22:51:34.588253 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.585948 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-sblc2" podStartSLOduration=24.059282869 podStartE2EDuration="26.585926484s" totalImagesPullingTime="2.526643615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:34.270214563 +0000 UTC m=+288.525662266" watchObservedRunningTime="2026-07-08 22:51:34.585926484 +0000 UTC m=+288.841374187" Jul 08 22:51:34.684916 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.684892 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j9692"] Jul 08 22:51:34.688045 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:34.688009 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f41cd53_0ea2_49fa_8677_683e3abf8150.slice/crio-ad4bb75d55e81335cf535ca573f94a1be2039326171546ec91786ccbea30edf4 WatchSource:0}: Error finding container ad4bb75d55e81335cf535ca573f94a1be2039326171546ec91786ccbea30edf4: Status 404 returned error can't find the container with id ad4bb75d55e81335cf535ca573f94a1be2039326171546ec91786ccbea30edf4 Jul 08 22:51:34.745388 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.745351 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-fdz8x"] Jul 08 22:51:34.750042 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.750013 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-fdz8x" Jul 08 22:51:34.753430 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.753187 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mb9hn\"" Jul 08 22:51:34.753588 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.753554 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 22:51:34.753796 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.753776 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 22:51:34.760024 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.759925 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-fdz8x"] Jul 08 22:51:34.766769 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.766730 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:34.854211 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.854171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:34.925288 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.925254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs5sc\" (UniqueName: \"kubernetes.io/projected/1ef9ab8d-16ec-4e0e-a553-87cee0316d49-kube-api-access-gs5sc\") pod \"downloads-685bfb5f64-fdz8x\" (UID: \"1ef9ab8d-16ec-4e0e-a553-87cee0316d49\") " pod="openshift-console/downloads-685bfb5f64-fdz8x" Jul 08 22:51:34.955011 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:34.954975 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:35.026368 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:35.026284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5sc\" (UniqueName: \"kubernetes.io/projected/1ef9ab8d-16ec-4e0e-a553-87cee0316d49-kube-api-access-gs5sc\") pod \"downloads-685bfb5f64-fdz8x\" (UID: \"1ef9ab8d-16ec-4e0e-a553-87cee0316d49\") " pod="openshift-console/downloads-685bfb5f64-fdz8x" Jul 08 22:51:35.034833 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:35.034803 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5sc\" (UniqueName: \"kubernetes.io/projected/1ef9ab8d-16ec-4e0e-a553-87cee0316d49-kube-api-access-gs5sc\") pod \"downloads-685bfb5f64-fdz8x\" (UID: \"1ef9ab8d-16ec-4e0e-a553-87cee0316d49\") " pod="openshift-console/downloads-685bfb5f64-fdz8x" Jul 08 22:51:35.054685 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:35.054650 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:35.063505 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:35.063479 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-fdz8x" Jul 08 22:51:35.220268 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:35.220233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:35.236189 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:35.236158 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j9692" event={"ID":"4f41cd53-0ea2-49fa-8677-683e3abf8150","Type":"ContainerStarted","Data":"ad4bb75d55e81335cf535ca573f94a1be2039326171546ec91786ccbea30edf4"} Jul 08 22:51:35.238152 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:35.238123 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xv6dw" event={"ID":"407a5c0c-24fd-474d-b677-65c32d5c1ca5","Type":"ContainerStarted","Data":"7073376c43da94e82650ae6c6aebd0bc0f54c04f22a3be2e1794e447f4d3bbc8"} Jul 08 22:51:35.238289 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:35.238173 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xv6dw" event={"ID":"407a5c0c-24fd-474d-b677-65c32d5c1ca5","Type":"ContainerStarted","Data":"99ba48518c48bf122121e95bbde0fd4c06855b302834dede2b412f0e8b2a817e"} Jul 08 22:51:35.238289 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:35.238214 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xv6dw" Jul 08 22:51:35.255821 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:35.255769 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xv6dw" podStartSLOduration=251.120656666 podStartE2EDuration="4m13.255751439s" totalImagesPullingTime="2.135094773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:35.254899092 +0000 UTC m=+289.510346821" watchObservedRunningTime="2026-07-08 22:51:35.255751439 +0000 UTC m=+289.511199326" Jul 08 22:51:35.318239 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:35.318161 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:35.641224 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:35.641137 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:35.742900 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:35.742846 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:35.974017 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:35.973959 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-fdz8x"] Jul 08 22:51:35.975954 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:35.975930 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef9ab8d_16ec_4e0e_a553_87cee0316d49.slice/crio-b83353ed606924d97bb53f529272cc5edf269ee072c857ec57c9b4ab0e5c9346 WatchSource:0}: Error finding container b83353ed606924d97bb53f529272cc5edf269ee072c857ec57c9b4ab0e5c9346: Status 404 returned error can't find the container with id b83353ed606924d97bb53f529272cc5edf269ee072c857ec57c9b4ab0e5c9346 Jul 08 22:51:36.044053 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:36.044014 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:36.145367 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:36.145281 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:36.242814 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:36.242779 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77fdr" event={"ID":"e1cc7e87-c580-4f51-8870-9ed1cbe58166","Type":"ContainerStarted","Data":"4218467f882c0e39c3110c722a9d42ebd646cfe505496549c88cc871e45feb93"} Jul 08 22:51:36.243814 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:36.243784 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-fdz8x" event={"ID":"1ef9ab8d-16ec-4e0e-a553-87cee0316d49","Type":"ContainerStarted","Data":"b83353ed606924d97bb53f529272cc5edf269ee072c857ec57c9b4ab0e5c9346"} Jul 08 22:51:36.245165 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:36.245138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:36.258692 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:36.258641 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-77fdr" podStartSLOduration=17.113503171 podStartE2EDuration="24.25862626s" totalImagesPullingTime="7.145123089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 22:51:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:36.257957828 +0000 UTC m=+290.513405540" watchObservedRunningTime="2026-07-08 22:51:36.25862626 +0000 UTC m=+290.514073963" Jul 08 22:51:36.346099 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:36.346047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:36.446446 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:36.446340 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:36.557198 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:36.557164 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:36.830865 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:36.830825 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:37.248221 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:37.248184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j9692" event={"ID":"4f41cd53-0ea2-49fa-8677-683e3abf8150","Type":"ContainerStarted","Data":"e5860e09964476f4cbc7111018b04b574775c6c75366df257cbea4028570ae7e"} Jul 08 22:51:37.248613 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:37.248448 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j9692" Jul 08 22:51:37.253056 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:37.253032 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j9692" Jul 08 22:51:37.263783 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:37.263746 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-j9692" podStartSLOduration=1.28049074 podStartE2EDuration="3.26372994s" totalImagesPullingTime="1.9832392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:37.262272867 +0000 UTC m=+291.517720581" watchObservedRunningTime="2026-07-08 22:51:37.26372994 +0000 UTC m=+291.519177644" Jul 08 22:51:38.231911 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:38.231871 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:38.295125 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:38.295073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:38.447943 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:38.447885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:40.212248 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:40.212209 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:42.350502 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.350464 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:42.569847 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.569813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:42.594852 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.594819 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls"] Jul 08 22:51:42.599644 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.599618 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" Jul 08 22:51:42.603389 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.603195 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ndsbd\"" Jul 08 22:51:42.603389 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.603185 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 08 22:51:42.603389 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.603251 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 08 22:51:42.603389 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.603341 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 22:51:42.610135 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.610106 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls"] Jul 08 22:51:42.618282 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.618259 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8"] Jul 08 22:51:42.621787 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.621766 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.625010 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.624989 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kst75\"" Jul 08 22:51:42.625203 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.625051 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 08 22:51:42.625282 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.625111 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 08 22:51:42.625341 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.625294 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k56fx"] Jul 08 22:51:42.625677 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.625655 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 08 22:51:42.628638 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.628620 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.630827 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.630733 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 22:51:42.631009 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.630970 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 22:51:42.631111 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.631020 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 22:51:42.631111 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.631048 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8"] Jul 08 22:51:42.631319 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.631303 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zr2jk\"" Jul 08 22:51:42.648516 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.648499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:42.686035 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.685999 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.686196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.686196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686095 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-wtmp\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.686196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686119 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhltv\" (UniqueName: \"kubernetes.io/projected/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-kube-api-access-fhltv\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.686196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686144 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.686196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-root\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.686196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686192 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-tls\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.686438 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.686438 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.686438 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-metrics-client-ca\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.686438 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.686438 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686412 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-textfile\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.686656 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686451 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5l5\" (UniqueName: \"kubernetes.io/projected/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-api-access-kd5l5\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.686656 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7tqr\" (UniqueName: \"kubernetes.io/projected/868a794a-7d4a-4c8c-82ed-e8cf93aecd4a-kube-api-access-l7tqr\") pod \"openshift-state-metrics-54cf96dfcf-tc2ls\" (UID: \"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" Jul 08 22:51:42.686656 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686511 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-accelerators-collector-config\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.686656 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686539 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/868a794a-7d4a-4c8c-82ed-e8cf93aecd4a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-tc2ls\" (UID: \"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" Jul 08 22:51:42.686656 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-sys\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.686656 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686595 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/868a794a-7d4a-4c8c-82ed-e8cf93aecd4a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-tc2ls\" (UID: \"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" Jul 08 22:51:42.686656 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.686622 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/868a794a-7d4a-4c8c-82ed-e8cf93aecd4a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-tc2ls\" (UID: \"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" Jul 08 22:51:42.713329 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.713304 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:42.787663 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.787629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.787832 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.787705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-root\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.787832 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.787736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-tls\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.787832 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.787779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.787832 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.787817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.788064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.787853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-metrics-client-ca\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.788064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.787886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.788064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.787920 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-textfile\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.788064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.787949 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5l5\" (UniqueName: \"kubernetes.io/projected/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-api-access-kd5l5\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.788064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.787977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7tqr\" (UniqueName: \"kubernetes.io/projected/868a794a-7d4a-4c8c-82ed-e8cf93aecd4a-kube-api-access-l7tqr\") pod \"openshift-state-metrics-54cf96dfcf-tc2ls\" (UID: \"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" Jul 08 22:51:42.788064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.788002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-accelerators-collector-config\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.788064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.788033 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/868a794a-7d4a-4c8c-82ed-e8cf93aecd4a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-tc2ls\" (UID: \"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" Jul 08 22:51:42.788770 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.788067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-sys\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.788770 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.788112 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/868a794a-7d4a-4c8c-82ed-e8cf93aecd4a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-tc2ls\" (UID: \"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" Jul 08 22:51:42.788770 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.788141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/868a794a-7d4a-4c8c-82ed-e8cf93aecd4a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-tc2ls\" (UID: \"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" Jul 08 22:51:42.788770 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:42.788382 2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 08 22:51:42.788770 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:42.788402 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 08 22:51:42.788770 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:42.788470 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-state-metrics-tls podName:5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:43.28844701 +0000 UTC m=+297.543894691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-zscp8" (UID: "5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9") : secret "kube-state-metrics-tls" not found Jul 08 22:51:42.788770 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:42.788492 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-tls podName:2c84f58f-ff9a-4aa2-bbcf-5ea023174a99 nodeName:}" failed. No retries permitted until 2026-07-08 22:51:43.288481589 +0000 UTC m=+297.543929273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-tls") pod "node-exporter-k56fx" (UID: "2c84f58f-ff9a-4aa2-bbcf-5ea023174a99") : secret "node-exporter-tls" not found Jul 08 22:51:42.788770 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.788530 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-textfile\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.788770 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.788577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.788770 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.788610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.788770 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.788650 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-wtmp\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.788770 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.788675 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhltv\" (UniqueName: \"kubernetes.io/projected/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-kube-api-access-fhltv\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.789457 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.788962 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-metrics-client-ca\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.789457 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.789029 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-root\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.789457 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.789029 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-sys\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.789894 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.789729 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.789894 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.789784 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.789894 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.789797 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.789894 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.789825 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/868a794a-7d4a-4c8c-82ed-e8cf93aecd4a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-tc2ls\" (UID: \"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" Jul 08 22:51:42.789894 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.789873 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-wtmp\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.790316 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.790259 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-accelerators-collector-config\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.791324 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.791302 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.791511 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.791469 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.791511 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.791481 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/868a794a-7d4a-4c8c-82ed-e8cf93aecd4a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-tc2ls\" (UID: \"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" Jul 08 22:51:42.791831 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.791812 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/868a794a-7d4a-4c8c-82ed-e8cf93aecd4a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-tc2ls\" (UID: \"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" Jul 08 22:51:42.799036 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.799007 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5l5\" (UniqueName: \"kubernetes.io/projected/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-api-access-kd5l5\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:42.800337 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.800296 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhltv\" (UniqueName: \"kubernetes.io/projected/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-kube-api-access-fhltv\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:42.800837 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.800796 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7tqr\" (UniqueName: \"kubernetes.io/projected/868a794a-7d4a-4c8c-82ed-e8cf93aecd4a-kube-api-access-l7tqr\") pod \"openshift-state-metrics-54cf96dfcf-tc2ls\" (UID: \"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" Jul 08 22:51:42.810984 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.810926 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:42.910229 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.910055 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:42.916927 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:42.916903 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" Jul 08 22:51:43.013847 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.013755 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:43.068861 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.068830 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls"] Jul 08 22:51:43.074465 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:43.072114 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868a794a_7d4a_4c8c_82ed_e8cf93aecd4a.slice/crio-b04b0ff05e8d6c61d748e5291588851f584752e683f7beda1d9365d641ee7da0 WatchSource:0}: Error finding container b04b0ff05e8d6c61d748e5291588851f584752e683f7beda1d9365d641ee7da0: Status 404 returned error can't find the container with id b04b0ff05e8d6c61d748e5291588851f584752e683f7beda1d9365d641ee7da0 Jul 08 22:51:43.077196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.077171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:43.272878 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.272846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" event={"ID":"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a","Type":"ContainerStarted","Data":"199ebe56389c5c8dd960adf11e3dd344f68a1a302ab130627dcfba0811d13e5c"} Jul 08 22:51:43.273009 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.272902 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" event={"ID":"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a","Type":"ContainerStarted","Data":"4c389abb8b72733e8abb5181538df710e7e58d80af0ff7e82f7dfde0d0f46cab"} Jul 08 22:51:43.273009 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.272924 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" event={"ID":"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a","Type":"ContainerStarted","Data":"b04b0ff05e8d6c61d748e5291588851f584752e683f7beda1d9365d641ee7da0"} Jul 08 22:51:43.294304 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.294281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:43.294409 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.294351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-tls\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:43.296615 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.296597 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2c84f58f-ff9a-4aa2-bbcf-5ea023174a99-node-exporter-tls\") pod \"node-exporter-k56fx\" (UID: \"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99\") " pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:43.296760 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.296743 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-zscp8\" (UID: \"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:43.532614 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.532578 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" Jul 08 22:51:43.539470 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.539443 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k56fx" Jul 08 22:51:43.553040 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:43.553006 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c84f58f_ff9a_4aa2_bbcf_5ea023174a99.slice/crio-ba34143caafb9f00529c1b2678b613f7d32c50a1af544f0d736d157072f08a2a WatchSource:0}: Error finding container ba34143caafb9f00529c1b2678b613f7d32c50a1af544f0d736d157072f08a2a: Status 404 returned error can't find the container with id ba34143caafb9f00529c1b2678b613f7d32c50a1af544f0d736d157072f08a2a Jul 08 22:51:43.567398 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.567366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:43.646015 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.645981 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:43.687726 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.687602 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8"] Jul 08 22:51:43.690131 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:43.690097 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef646bd_7f85_4542_b4b9_b8c5cbcc5fd9.slice/crio-d8fe9ea47ad3cc2869f74d0664014724da9859164ca01fde631e0a5f46ea9efe WatchSource:0}: Error finding container d8fe9ea47ad3cc2869f74d0664014724da9859164ca01fde631e0a5f46ea9efe: Status 404 returned error can't find the container with id d8fe9ea47ad3cc2869f74d0664014724da9859164ca01fde631e0a5f46ea9efe Jul 08 22:51:43.768741 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:43.768711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:44.262271 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:51:44.262224 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-rldqz" podUID="4ed6eae7-7b43-47ba-b489-136fab2a8ade" Jul 08 22:51:44.279454 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:44.279375 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" event={"ID":"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9","Type":"ContainerStarted","Data":"d8fe9ea47ad3cc2869f74d0664014724da9859164ca01fde631e0a5f46ea9efe"} Jul 08 22:51:44.281468 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:44.281396 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k56fx" event={"ID":"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99","Type":"ContainerStarted","Data":"ba34143caafb9f00529c1b2678b613f7d32c50a1af544f0d736d157072f08a2a"} Jul 08 22:51:45.246345 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.246321 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xv6dw" Jul 08 22:51:45.483125 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.483023 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dd57b4cf4-9vhsl"] Jul 08 22:51:45.486258 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.486217 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.489067 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.488835 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bh65l\"" Jul 08 22:51:45.489067 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.488975 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 22:51:45.489310 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.489239 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 22:51:45.489310 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.489267 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 22:51:45.489536 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.489511 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 22:51:45.489770 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.489751 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 22:51:45.496716 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.496693 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dd57b4cf4-9vhsl"] Jul 08 22:51:45.516264 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.516240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-service-ca\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.516510 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.516470 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq6zx\" (UniqueName: \"kubernetes.io/projected/c3285330-1d1e-4d21-a032-df1a01f22bfc-kube-api-access-cq6zx\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.516708 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.516673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-serving-cert\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.516878 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.516853 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-oauth-config\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.517073 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.517055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-oauth-serving-cert\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.517253 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.517237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-config\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.557403 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.557370 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:45.581336 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.581279 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76d47b86df-n7r7k"] Jul 08 22:51:45.587401 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.587380 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.590253 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.590013 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 08 22:51:45.590253 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.590053 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 08 22:51:45.590253 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.590064 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8fd9aekn08ors\"" Jul 08 22:51:45.590253 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.590014 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 08 22:51:45.590253 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.590222 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ns7df\"" Jul 08 22:51:45.590603 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.590420 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 08 22:51:45.590729 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.590708 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 08 22:51:45.596414 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.596333 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76d47b86df-n7r7k"] Jul 08 22:51:45.618244 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.618211 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n7kb\" (UniqueName: \"kubernetes.io/projected/ad17e873-f56b-4bf4-a664-2eff93d0aecb-kube-api-access-2n7kb\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.618385 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.618260 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-serving-cert\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.618385 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.618289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.618385 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.618325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-oauth-config\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.618385 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.618348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-tls\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.618634 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.618450 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.618634 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.618482 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-grpc-tls\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.618634 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.618535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-oauth-serving-cert\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.618634 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.618565 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-config\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.618634 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.618593 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.618865 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.618652 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.618865 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.618730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-service-ca\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.618865 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.618770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad17e873-f56b-4bf4-a664-2eff93d0aecb-metrics-client-ca\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.618865 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.618800 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq6zx\" (UniqueName: \"kubernetes.io/projected/c3285330-1d1e-4d21-a032-df1a01f22bfc-kube-api-access-cq6zx\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.619365 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.619339 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-oauth-serving-cert\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.619597 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.619467 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-config\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.619707 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.619609 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-service-ca\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.621201 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.621181 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-oauth-config\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.621286 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.621223 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-serving-cert\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.626476 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.626456 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq6zx\" (UniqueName: \"kubernetes.io/projected/c3285330-1d1e-4d21-a032-df1a01f22bfc-kube-api-access-cq6zx\") pod \"console-dd57b4cf4-9vhsl\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.658626 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.658598 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:45.719459 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.719386 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n7kb\" (UniqueName: \"kubernetes.io/projected/ad17e873-f56b-4bf4-a664-2eff93d0aecb-kube-api-access-2n7kb\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.719459 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.719423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.719459 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.719449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-tls\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.719729 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.719471 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.719729 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.719504 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-grpc-tls\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.719729 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.719535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.719729 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.719578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.719729 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.719638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad17e873-f56b-4bf4-a664-2eff93d0aecb-metrics-client-ca\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.720431 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.720369 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad17e873-f56b-4bf4-a664-2eff93d0aecb-metrics-client-ca\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.722674 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.722646 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.722780 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.722673 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.722780 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.722756 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-tls\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.722864 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.722837 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-grpc-tls\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.723071 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.723052 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.723163 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.723146 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ad17e873-f56b-4bf4-a664-2eff93d0aecb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.726822 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.726804 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n7kb\" (UniqueName: \"kubernetes.io/projected/ad17e873-f56b-4bf4-a664-2eff93d0aecb-kube-api-access-2n7kb\") pod \"thanos-querier-76d47b86df-n7r7k\" (UID: \"ad17e873-f56b-4bf4-a664-2eff93d0aecb\") " pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:45.759855 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.759825 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:45.811718 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.811679 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:51:45.904119 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:45.902328 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:51:46.009485 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.009328 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dd57b4cf4-9vhsl"] Jul 08 22:51:46.012619 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:46.012586 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3285330_1d1e_4d21_a032_df1a01f22bfc.slice/crio-932c08edcbfb2474417505791f0560bfd1a2eb4f15fe9e80e243d81cbb0f899f WatchSource:0}: Error finding container 932c08edcbfb2474417505791f0560bfd1a2eb4f15fe9e80e243d81cbb0f899f: Status 404 returned error can't find the container with id 932c08edcbfb2474417505791f0560bfd1a2eb4f15fe9e80e243d81cbb0f899f Jul 08 22:51:46.072584 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.072442 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76d47b86df-n7r7k"] Jul 08 22:51:46.074921 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:46.074891 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad17e873_f56b_4bf4_a664_2eff93d0aecb.slice/crio-64699104cefedac26f0a3b8039f88e16b3fc12036e60328b2bd8853cfca8a872 WatchSource:0}: Error finding container 64699104cefedac26f0a3b8039f88e16b3fc12036e60328b2bd8853cfca8a872: Status 404 returned error can't find the container with id 64699104cefedac26f0a3b8039f88e16b3fc12036e60328b2bd8853cfca8a872 Jul 08 22:51:46.076760 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.076731 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:46.171967 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.171937 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 22:51:46.172117 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.171939 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 22:51:46.175694 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.175675 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 22:51:46.175803 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.175678 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 22:51:46.293438 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.293290 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" event={"ID":"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9","Type":"ContainerStarted","Data":"6184535f5841342030aebf656824b29443888324f88bff6a57025a7115097cad"} Jul 08 22:51:46.293438 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.293353 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" event={"ID":"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9","Type":"ContainerStarted","Data":"3b79996b75ff0fae60081a8df4b7a63ee5cd1e208f5c00e253ca2fd9c9f90a0f"} Jul 08 22:51:46.293438 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.293380 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" event={"ID":"5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9","Type":"ContainerStarted","Data":"fdd5dc155ec4f8065f8a0f7a249f3d84eba4d596fe3523a0716c811314d16099"} Jul 08 22:51:46.676512 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.296036 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" event={"ID":"868a794a-7d4a-4c8c-82ed-e8cf93aecd4a","Type":"ContainerStarted","Data":"b89bd4186db4db031c3b991a6117bd24b14851bccaf6d4299b3fca3f89e61f1c"} Jul 08 22:51:46.676512 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.297586 2564 generic.go:356] "Generic (PLEG): container finished" podID="2c84f58f-ff9a-4aa2-bbcf-5ea023174a99" containerID="6d79aa3203eb71d4c386f45a0d3168b4a40c3ade9138fb71576bda2d88c0d129" exitCode=0 Jul 08 22:51:46.676512 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.297679 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k56fx" event={"ID":"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99","Type":"ContainerDied","Data":"6d79aa3203eb71d4c386f45a0d3168b4a40c3ade9138fb71576bda2d88c0d129"} Jul 08 22:51:46.676512 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.298966 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" event={"ID":"ad17e873-f56b-4bf4-a664-2eff93d0aecb","Type":"ContainerStarted","Data":"64699104cefedac26f0a3b8039f88e16b3fc12036e60328b2bd8853cfca8a872"} Jul 08 22:51:46.676512 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.300215 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd57b4cf4-9vhsl" event={"ID":"c3285330-1d1e-4d21-a032-df1a01f22bfc","Type":"ContainerStarted","Data":"932c08edcbfb2474417505791f0560bfd1a2eb4f15fe9e80e243d81cbb0f899f"} Jul 08 22:51:46.676512 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.311008 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zscp8" podStartSLOduration=2.787419486 podStartE2EDuration="4.310990054s" totalImagesPullingTime="1.523570568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:46.308568285 +0000 UTC m=+300.564015989" watchObservedRunningTime="2026-07-08 22:51:46.310990054 +0000 UTC m=+300.566437758" Jul 08 22:51:46.676512 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.325767 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-tc2ls" podStartSLOduration=2.326697012 podStartE2EDuration="4.325755372s" totalImagesPullingTime="1.99905836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:46.324531222 +0000 UTC m=+300.579978927" watchObservedRunningTime="2026-07-08 22:51:46.325755372 +0000 UTC m=+300.581203074" Jul 08 22:51:46.997719 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:46.997686 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-587895c54b-tfndr"] Jul 08 22:51:47.020564 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.020530 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-587895c54b-tfndr"] Jul 08 22:51:47.020726 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.020658 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.029711 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.029672 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 08 22:51:47.030649 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.029940 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 08 22:51:47.030649 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.030148 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 08 22:51:47.031525 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.030909 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dlajanpm1gmvt\"" Jul 08 22:51:47.031525 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.031141 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 08 22:51:47.031525 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.031326 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8r54c\"" Jul 08 22:51:47.135026 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.134986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/26e7d046-8455-4ac9-969f-c436e90f8520-secret-metrics-server-client-certs\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.135200 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.135033 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/26e7d046-8455-4ac9-969f-c436e90f8520-audit-log\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.135200 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.135060 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/26e7d046-8455-4ac9-969f-c436e90f8520-client-ca-bundle\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.135200 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.135115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/26e7d046-8455-4ac9-969f-c436e90f8520-metrics-server-audit-profiles\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.135200 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.135153 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e7d046-8455-4ac9-969f-c436e90f8520-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.135331 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.135301 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl9c2\" (UniqueName: \"kubernetes.io/projected/26e7d046-8455-4ac9-969f-c436e90f8520-kube-api-access-kl9c2\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.135370 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.135350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/26e7d046-8455-4ac9-969f-c436e90f8520-secret-metrics-server-tls\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.237116 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.236631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl9c2\" (UniqueName: \"kubernetes.io/projected/26e7d046-8455-4ac9-969f-c436e90f8520-kube-api-access-kl9c2\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.237116 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.236688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/26e7d046-8455-4ac9-969f-c436e90f8520-secret-metrics-server-tls\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.237116 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.236748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/26e7d046-8455-4ac9-969f-c436e90f8520-secret-metrics-server-client-certs\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.237116 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.236773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/26e7d046-8455-4ac9-969f-c436e90f8520-audit-log\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.237116 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.236798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/26e7d046-8455-4ac9-969f-c436e90f8520-client-ca-bundle\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.237116 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.236830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/26e7d046-8455-4ac9-969f-c436e90f8520-metrics-server-audit-profiles\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.237116 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.237054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e7d046-8455-4ac9-969f-c436e90f8520-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.237588 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.237255 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/26e7d046-8455-4ac9-969f-c436e90f8520-audit-log\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.237857 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.237810 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e7d046-8455-4ac9-969f-c436e90f8520-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.238218 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.238187 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/26e7d046-8455-4ac9-969f-c436e90f8520-metrics-server-audit-profiles\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.242861 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.242066 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/26e7d046-8455-4ac9-969f-c436e90f8520-secret-metrics-server-tls\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.246469 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.243642 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/26e7d046-8455-4ac9-969f-c436e90f8520-client-ca-bundle\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.246469 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.246301 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl9c2\" (UniqueName: \"kubernetes.io/projected/26e7d046-8455-4ac9-969f-c436e90f8520-kube-api-access-kl9c2\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.247369 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.247307 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/26e7d046-8455-4ac9-969f-c436e90f8520-secret-metrics-server-client-certs\") pod \"metrics-server-587895c54b-tfndr\" (UID: \"26e7d046-8455-4ac9-969f-c436e90f8520\") " pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.308397 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.308288 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k56fx" event={"ID":"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99","Type":"ContainerStarted","Data":"735288a9b1f28b525a92ac7f3503e5c793069ef8f6355df4743cb8a77feebb8a"} Jul 08 22:51:47.308960 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.308779 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k56fx" event={"ID":"2c84f58f-ff9a-4aa2-bbcf-5ea023174a99","Type":"ContainerStarted","Data":"d6eb845ca85cfb9fbff62c00738211b3850288437282d48337a3a50d55b7bf8d"} Jul 08 22:51:47.328122 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.327291 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k56fx" podStartSLOduration=3.557004773 podStartE2EDuration="5.327269889s" totalImagesPullingTime="1.657827699s" totalInitContainerRuntime="112.437417ms" isStatefulPod=true podCreationTimestamp="2026-07-08 22:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:47.325924918 +0000 UTC m=+301.581372621" watchObservedRunningTime="2026-07-08 22:51:47.327269889 +0000 UTC m=+301.582717593" Jul 08 22:51:47.341655 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.341434 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:51:47.373559 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.373529 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:47.429229 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.429196 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:47.510667 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.510588 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-587895c54b-tfndr"] Jul 08 22:51:47.515073 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:47.515038 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e7d046_8455_4ac9_969f_c436e90f8520.slice/crio-4f66d2cf427f71bc951cad4e0e20e9aa6ca197c697e9f24adcb49089a6f8d57b WatchSource:0}: Error finding container 4f66d2cf427f71bc951cad4e0e20e9aa6ca197c697e9f24adcb49089a6f8d57b: Status 404 returned error can't find the container with id 4f66d2cf427f71bc951cad4e0e20e9aa6ca197c697e9f24adcb49089a6f8d57b Jul 08 22:51:47.709475 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:47.709368 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:48.314412 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.314366 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-587895c54b-tfndr" event={"ID":"26e7d046-8455-4ac9-969f-c436e90f8520","Type":"ContainerStarted","Data":"4f66d2cf427f71bc951cad4e0e20e9aa6ca197c697e9f24adcb49089a6f8d57b"} Jul 08 22:51:48.833750 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.833716 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 22:51:48.835393 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.835362 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:48.836977 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.836948 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.839549 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.839525 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 08 22:51:48.839635 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.839617 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 08 22:51:48.839635 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.839626 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 08 22:51:48.839757 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.839742 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 08 22:51:48.839949 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.839856 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 08 22:51:48.839949 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.839911 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 08 22:51:48.840217 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.840046 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 08 22:51:48.840217 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.840059 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 08 22:51:48.840217 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.840126 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-66664lm5btme8\"" Jul 08 22:51:48.840217 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.840151 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 08 22:51:48.840592 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.840233 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 08 22:51:48.840592 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.840491 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 08 22:51:48.840592 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.840556 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 08 22:51:48.840830 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.840811 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 08 22:51:48.840924 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.840845 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nh89h\"" Jul 08 22:51:48.843175 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.843156 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 08 22:51:48.854375 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.854352 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 22:51:48.955532 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.955498 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.955709 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.955538 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-web-config\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.955709 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.955575 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.955709 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.955631 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.955709 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.955703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.955871 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.955732 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.955871 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.955763 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.955871 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.955788 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.955871 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.955833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.955871 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.955856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.956126 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.955885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-config\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.956126 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.955906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.956126 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.955920 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42468\" (UniqueName: \"kubernetes.io/projected/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-kube-api-access-42468\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.956126 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.955947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.956126 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.956042 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-config-out\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.956359 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.956112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.956359 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.956158 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.956359 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.956188 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.956359 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.956212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:48.956359 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:48.956243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057330 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057293 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057527 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057527 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057527 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057527 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057418 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057527 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057527 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057527 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057497 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-config\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057527 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057940 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057544 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42468\" (UniqueName: \"kubernetes.io/projected/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-kube-api-access-42468\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057940 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057581 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057940 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-config-out\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057940 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057940 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057940 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057940 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057940 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057778 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057940 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057814 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057940 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-web-config\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.057940 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.057869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.058513 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.058226 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.058513 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.058466 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.058620 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.058514 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.058746 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.058717 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.061117 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.058973 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.061117 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.059798 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.061117 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.061023 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.061386 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.061358 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.061984 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.061519 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.061984 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.061742 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-config\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.061984 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.061945 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.062225 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.062129 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.062836 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.062812 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.062941 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.062865 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.063502 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.063480 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-config-out\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.064653 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.064628 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.064760 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.064633 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.064883 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.064856 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-web-config\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.064883 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.064873 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.066147 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.066126 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42468\" (UniqueName: \"kubernetes.io/projected/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-kube-api-access-42468\") pod \"prometheus-k8s-0\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.149127 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.149018 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:51:49.970139 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:49.970104 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:50.401091 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:50.401046 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:50.448847 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:50.448813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:52.220229 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:52.220186 2564 patch_prober.go:28] interesting pod/image-registry-594fb948ff-jktv6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 08 22:51:52.220612 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:52.220258 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" podUID="82dddf27-89f2-45f8-ba41-74a7be616df4" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 22:51:52.463242 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:52.463181 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:52.563556 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:52.563518 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:52.665585 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:52.665556 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:53.662399 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:53.662367 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:53.771901 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:53.771863 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:54.235936 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:54.235902 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:51:54.359260 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:54.359229 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:54.460366 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:54.460324 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:54.562275 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:54.562236 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:57.120718 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.120688 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:57.221227 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.221199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:57.261912 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.261839 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:51:57.313401 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.313303 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 22:51:57.315642 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:51:57.315615 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffb1519_4d3d_4c5d_9149_32c6b0d49fbe.slice/crio-5b0c8533a00967eaf10c837ca6fbd9a19ffe35458a9a6c87983ca4d25b34a6cf WatchSource:0}: Error finding container 5b0c8533a00967eaf10c837ca6fbd9a19ffe35458a9a6c87983ca4d25b34a6cf: Status 404 returned error can't find the container with id 5b0c8533a00967eaf10c837ca6fbd9a19ffe35458a9a6c87983ca4d25b34a6cf Jul 08 22:51:57.319252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.317953 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 22:51:57.361335 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.360710 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" event={"ID":"ad17e873-f56b-4bf4-a664-2eff93d0aecb","Type":"ContainerStarted","Data":"3880a2a9a3f1eb46c6d206707419bf3c282d01c79816baa5398a6c00ee5d694b"} Jul 08 22:51:57.361335 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.360792 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" event={"ID":"ad17e873-f56b-4bf4-a664-2eff93d0aecb","Type":"ContainerStarted","Data":"6c0ed737af9d26857727b1b43a7d9286d6c904d72dade59bb8f5c880fc93a440"} Jul 08 22:51:57.364472 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.364103 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd57b4cf4-9vhsl" event={"ID":"c3285330-1d1e-4d21-a032-df1a01f22bfc","Type":"ContainerStarted","Data":"6943704e15646b4fc1909625d437ba26f30b2689c69df75971d7c89e84e0b227"} Jul 08 22:51:57.367561 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.367528 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerStarted","Data":"5b0c8533a00967eaf10c837ca6fbd9a19ffe35458a9a6c87983ca4d25b34a6cf"} Jul 08 22:51:57.369820 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.369655 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-fdz8x" event={"ID":"1ef9ab8d-16ec-4e0e-a553-87cee0316d49","Type":"ContainerStarted","Data":"bb85985f158566e5dbcaf0f6beacdc6585aff7082ac65b30a056c33c2bd1284e"} Jul 08 22:51:57.370782 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.370216 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-fdz8x" Jul 08 22:51:57.372227 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.371946 2564 patch_prober.go:28] interesting pod/downloads-685bfb5f64-fdz8x container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.18:8080/\": dial tcp 10.132.0.18:8080: connect: connection refused" start-of-body= Jul 08 22:51:57.372227 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.371990 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-fdz8x" podUID="1ef9ab8d-16ec-4e0e-a553-87cee0316d49" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.18:8080/\": dial tcp 10.132.0.18:8080: connect: connection refused" Jul 08 22:51:57.399948 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.399884 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-fdz8x" podStartSLOduration=2.181727918 podStartE2EDuration="23.399861721s" totalImagesPullingTime="21.218133803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:57.397681229 +0000 UTC m=+311.653128934" watchObservedRunningTime="2026-07-08 22:51:57.399861721 +0000 UTC m=+311.655309427" Jul 08 22:51:57.400158 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.400015 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dd57b4cf4-9vhsl" podStartSLOduration=1.255491701 podStartE2EDuration="12.400003365s" totalImagesPullingTime="11.144511664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:51:57.381741759 +0000 UTC m=+311.637189497" watchObservedRunningTime="2026-07-08 22:51:57.400003365 +0000 UTC m=+311.655451071" Jul 08 22:51:57.450766 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.450734 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dd57b4cf4-9vhsl"] Jul 08 22:51:57.571307 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.570695 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-594fb948ff-jktv6"] Jul 08 22:51:57.640191 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.640057 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:57.778253 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.778219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:57.832725 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:57.832651 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:58.378453 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:58.378333 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" event={"ID":"ad17e873-f56b-4bf4-a664-2eff93d0aecb","Type":"ContainerStarted","Data":"976486a32c6df5f16434af6c431b50b8fcca3c2d12a45bdddc55fc4063bfb1c3"} Jul 08 22:51:58.394460 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:58.394422 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-fdz8x" Jul 08 22:51:59.531671 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:59.531640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:51:59.588059 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:51:59.588019 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:52:00.392301 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:00.392212 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" event={"ID":"ad17e873-f56b-4bf4-a664-2eff93d0aecb","Type":"ContainerStarted","Data":"77117477b62ed02437d8257c358169a41bfc2c74a936a767f1c0602afd01853f"} Jul 08 22:52:00.392436 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:00.392303 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" event={"ID":"ad17e873-f56b-4bf4-a664-2eff93d0aecb","Type":"ContainerStarted","Data":"fe42c65d797951ca6a07c6283d8a06767d979b13aa4fd08b8180af23dbb7d8e6"} Jul 08 22:52:00.392436 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:00.392335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" event={"ID":"ad17e873-f56b-4bf4-a664-2eff93d0aecb","Type":"ContainerStarted","Data":"2bed21214e732d80b412242b2bd1d00aacd67abbf286c6e2e7ea58b7f107cc3c"} Jul 08 22:52:00.392620 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:00.392597 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:52:00.394332 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:00.394304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-587895c54b-tfndr" event={"ID":"26e7d046-8455-4ac9-969f-c436e90f8520","Type":"ContainerStarted","Data":"98268b524a500a3dfbb296fe153e1a13701290af78e3f304812572bdf7cf8962"} Jul 08 22:52:00.396261 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:00.396232 2564 generic.go:356] "Generic (PLEG): container finished" podID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerID="6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf" exitCode=0 Jul 08 22:52:00.396261 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:00.396259 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerDied","Data":"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf"} Jul 08 22:52:00.414847 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:00.414786 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" podStartSLOduration=1.9629177549999999 podStartE2EDuration="15.414767249s" totalImagesPullingTime="13.451849494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:51:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:52:00.41337564 +0000 UTC m=+314.668823347" watchObservedRunningTime="2026-07-08 22:52:00.414767249 +0000 UTC m=+314.670214954" Jul 08 22:52:00.431749 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:00.431689 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-587895c54b-tfndr" podStartSLOduration=2.143287399 podStartE2EDuration="14.431669337s" totalImagesPullingTime="12.288381938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:52:00.43029899 +0000 UTC m=+314.685746693" watchObservedRunningTime="2026-07-08 22:52:00.431669337 +0000 UTC m=+314.687117040" Jul 08 22:52:03.124604 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:03.124556 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:52:03.225725 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:03.225686 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:52:05.430256 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:05.430226 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerStarted","Data":"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed"} Jul 08 22:52:05.430624 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:05.430311 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerStarted","Data":"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3"} Jul 08 22:52:05.430624 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:05.430358 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerStarted","Data":"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d"} Jul 08 22:52:05.430624 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:05.430399 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerStarted","Data":"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56"} Jul 08 22:52:05.430624 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:05.430439 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerStarted","Data":"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8"} Jul 08 22:52:05.430624 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:05.430481 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerStarted","Data":"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c"} Jul 08 22:52:05.812679 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:05.812643 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:52:06.410313 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:06.410283 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76d47b86df-n7r7k" Jul 08 22:52:06.464800 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:06.464713 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=11.704395903 podStartE2EDuration="18.464691327s" totalImagesPullingTime="6.737462869s" totalInitContainerRuntime="22.832555ms" isStatefulPod=false podCreationTimestamp="2026-07-08 22:51:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:52:06.463337566 +0000 UTC m=+320.718785271" watchObservedRunningTime="2026-07-08 22:52:06.464691327 +0000 UTC m=+320.720139037" Jul 08 22:52:07.342018 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:07.341984 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:52:07.342228 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:07.342066 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:52:08.750788 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:08.750760 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:52:08.853980 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:08.853944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:52:08.955305 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:08.955257 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:52:09.149705 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:09.149629 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:52:11.667723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:11.667691 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:52:11.768564 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:11.768533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:52:15.259902 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:15.259868 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:52:15.361217 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:15.361190 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:52:20.676597 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:20.676566 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:52:22.600107 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.600047 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" podUID="82dddf27-89f2-45f8-ba41-74a7be616df4" containerName="registry" containerID="cri-o://1b92a9584be49ed84db4ee3134d4ac8c6b4e9575a3f571d7c1745380390014a0" gracePeriod=30 Jul 08 22:52:22.871611 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.871546 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:52:22.983550 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.983520 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82dddf27-89f2-45f8-ba41-74a7be616df4-installation-pull-secrets\") pod \"82dddf27-89f2-45f8-ba41-74a7be616df4\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " Jul 08 22:52:22.983722 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.983567 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82dddf27-89f2-45f8-ba41-74a7be616df4-ca-trust-extracted\") pod \"82dddf27-89f2-45f8-ba41-74a7be616df4\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " Jul 08 22:52:22.983722 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.983605 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-bound-sa-token\") pod \"82dddf27-89f2-45f8-ba41-74a7be616df4\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " Jul 08 22:52:22.983841 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.983724 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-certificates\") pod \"82dddf27-89f2-45f8-ba41-74a7be616df4\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " Jul 08 22:52:22.983841 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.983786 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls\") pod \"82dddf27-89f2-45f8-ba41-74a7be616df4\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " Jul 08 22:52:22.983841 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.983827 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82dddf27-89f2-45f8-ba41-74a7be616df4-trusted-ca\") pod \"82dddf27-89f2-45f8-ba41-74a7be616df4\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " Jul 08 22:52:22.983992 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.983872 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs22h\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-kube-api-access-bs22h\") pod \"82dddf27-89f2-45f8-ba41-74a7be616df4\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " Jul 08 22:52:22.983992 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.983927 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82dddf27-89f2-45f8-ba41-74a7be616df4-image-registry-private-configuration\") pod \"82dddf27-89f2-45f8-ba41-74a7be616df4\" (UID: \"82dddf27-89f2-45f8-ba41-74a7be616df4\") " Jul 08 22:52:22.984289 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.984240 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "82dddf27-89f2-45f8-ba41-74a7be616df4" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 22:52:22.984632 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.984598 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82dddf27-89f2-45f8-ba41-74a7be616df4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "82dddf27-89f2-45f8-ba41-74a7be616df4" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 22:52:22.986141 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.986056 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "82dddf27-89f2-45f8-ba41-74a7be616df4" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 22:52:22.986141 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.986133 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "82dddf27-89f2-45f8-ba41-74a7be616df4" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 22:52:22.986308 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.986250 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82dddf27-89f2-45f8-ba41-74a7be616df4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "82dddf27-89f2-45f8-ba41-74a7be616df4" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 22:52:22.986520 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.986492 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-kube-api-access-bs22h" (OuterVolumeSpecName: "kube-api-access-bs22h") pod "82dddf27-89f2-45f8-ba41-74a7be616df4" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4"). InnerVolumeSpecName "kube-api-access-bs22h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 22:52:22.986591 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.986546 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82dddf27-89f2-45f8-ba41-74a7be616df4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "82dddf27-89f2-45f8-ba41-74a7be616df4" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 22:52:22.992319 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:22.992297 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dddf27-89f2-45f8-ba41-74a7be616df4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "82dddf27-89f2-45f8-ba41-74a7be616df4" (UID: "82dddf27-89f2-45f8-ba41-74a7be616df4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 22:52:23.085377 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.085342 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82dddf27-89f2-45f8-ba41-74a7be616df4-ca-trust-extracted\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:52:23.085377 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.085369 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-bound-sa-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:52:23.085377 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.085379 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-certificates\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:52:23.085377 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.085388 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-registry-tls\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:52:23.085724 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.085399 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82dddf27-89f2-45f8-ba41-74a7be616df4-trusted-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:52:23.085724 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.085406 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs22h\" (UniqueName: \"kubernetes.io/projected/82dddf27-89f2-45f8-ba41-74a7be616df4-kube-api-access-bs22h\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:52:23.085724 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.085415 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82dddf27-89f2-45f8-ba41-74a7be616df4-image-registry-private-configuration\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:52:23.085724 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.085423 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82dddf27-89f2-45f8-ba41-74a7be616df4-installation-pull-secrets\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:52:23.504279 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.504248 2564 generic.go:356] "Generic (PLEG): container finished" podID="82dddf27-89f2-45f8-ba41-74a7be616df4" containerID="1b92a9584be49ed84db4ee3134d4ac8c6b4e9575a3f571d7c1745380390014a0" exitCode=0 Jul 08 22:52:23.504464 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.504294 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" Jul 08 22:52:23.504464 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.504335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" event={"ID":"82dddf27-89f2-45f8-ba41-74a7be616df4","Type":"ContainerDied","Data":"1b92a9584be49ed84db4ee3134d4ac8c6b4e9575a3f571d7c1745380390014a0"} Jul 08 22:52:23.504464 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.504391 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-594fb948ff-jktv6" event={"ID":"82dddf27-89f2-45f8-ba41-74a7be616df4","Type":"ContainerDied","Data":"5e74f25f0a303986cdebcab341c6e88ccf8163b9a42a194f341bafcbb5ad9172"} Jul 08 22:52:23.504464 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.504417 2564 scope.go:117] "RemoveContainer" containerID="1b92a9584be49ed84db4ee3134d4ac8c6b4e9575a3f571d7c1745380390014a0" Jul 08 22:52:23.521855 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.521841 2564 scope.go:117] "RemoveContainer" containerID="1b92a9584be49ed84db4ee3134d4ac8c6b4e9575a3f571d7c1745380390014a0" Jul 08 22:52:23.522135 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:52:23.522119 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b92a9584be49ed84db4ee3134d4ac8c6b4e9575a3f571d7c1745380390014a0\": container with ID starting with 1b92a9584be49ed84db4ee3134d4ac8c6b4e9575a3f571d7c1745380390014a0 not found: ID does not exist" containerID="1b92a9584be49ed84db4ee3134d4ac8c6b4e9575a3f571d7c1745380390014a0" Jul 08 22:52:23.522198 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.522142 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b92a9584be49ed84db4ee3134d4ac8c6b4e9575a3f571d7c1745380390014a0"} err="failed to get container status \"1b92a9584be49ed84db4ee3134d4ac8c6b4e9575a3f571d7c1745380390014a0\": rpc error: code = NotFound desc = could not find container \"1b92a9584be49ed84db4ee3134d4ac8c6b4e9575a3f571d7c1745380390014a0\": container with ID starting with 1b92a9584be49ed84db4ee3134d4ac8c6b4e9575a3f571d7c1745380390014a0 not found: ID does not exist" Jul 08 22:52:23.524651 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.524632 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-594fb948ff-jktv6"] Jul 08 22:52:23.527858 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:23.527841 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-594fb948ff-jktv6"] Jul 08 22:52:24.266482 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.266451 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82dddf27-89f2-45f8-ba41-74a7be616df4" path="/var/lib/kubelet/pods/82dddf27-89f2-45f8-ba41-74a7be616df4/volumes" Jul 08 22:52:24.408396 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.408357 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-dd57b4cf4-9vhsl" podUID="c3285330-1d1e-4d21-a032-df1a01f22bfc" containerName="console" containerID="cri-o://6943704e15646b4fc1909625d437ba26f30b2689c69df75971d7c89e84e0b227" gracePeriod=15 Jul 08 22:52:24.697917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.697886 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:52:24.798917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.798845 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-config\") pod \"c3285330-1d1e-4d21-a032-df1a01f22bfc\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " Jul 08 22:52:24.798917 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.798898 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-oauth-config\") pod \"c3285330-1d1e-4d21-a032-df1a01f22bfc\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " Jul 08 22:52:24.799154 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.798919 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-oauth-serving-cert\") pod \"c3285330-1d1e-4d21-a032-df1a01f22bfc\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " Jul 08 22:52:24.799154 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.798949 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-serving-cert\") pod \"c3285330-1d1e-4d21-a032-df1a01f22bfc\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " Jul 08 22:52:24.799154 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.798999 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-service-ca\") pod \"c3285330-1d1e-4d21-a032-df1a01f22bfc\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " Jul 08 22:52:24.799154 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.799055 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq6zx\" (UniqueName: \"kubernetes.io/projected/c3285330-1d1e-4d21-a032-df1a01f22bfc-kube-api-access-cq6zx\") pod \"c3285330-1d1e-4d21-a032-df1a01f22bfc\" (UID: \"c3285330-1d1e-4d21-a032-df1a01f22bfc\") " Jul 08 22:52:24.799357 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.799336 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c3285330-1d1e-4d21-a032-df1a01f22bfc" (UID: "c3285330-1d1e-4d21-a032-df1a01f22bfc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 22:52:24.799407 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.799347 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-config" (OuterVolumeSpecName: "console-config") pod "c3285330-1d1e-4d21-a032-df1a01f22bfc" (UID: "c3285330-1d1e-4d21-a032-df1a01f22bfc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 22:52:24.799407 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.799393 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-service-ca" (OuterVolumeSpecName: "service-ca") pod "c3285330-1d1e-4d21-a032-df1a01f22bfc" (UID: "c3285330-1d1e-4d21-a032-df1a01f22bfc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 22:52:24.801215 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.801186 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c3285330-1d1e-4d21-a032-df1a01f22bfc" (UID: "c3285330-1d1e-4d21-a032-df1a01f22bfc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 22:52:24.801323 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.801224 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c3285330-1d1e-4d21-a032-df1a01f22bfc" (UID: "c3285330-1d1e-4d21-a032-df1a01f22bfc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 22:52:24.801323 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.801284 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3285330-1d1e-4d21-a032-df1a01f22bfc-kube-api-access-cq6zx" (OuterVolumeSpecName: "kube-api-access-cq6zx") pod "c3285330-1d1e-4d21-a032-df1a01f22bfc" (UID: "c3285330-1d1e-4d21-a032-df1a01f22bfc"). InnerVolumeSpecName "kube-api-access-cq6zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 22:52:24.900525 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.900494 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:52:24.900525 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.900521 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq6zx\" (UniqueName: \"kubernetes.io/projected/c3285330-1d1e-4d21-a032-df1a01f22bfc-kube-api-access-cq6zx\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:52:24.900525 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.900531 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-config\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:52:24.900742 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.900539 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-oauth-config\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:52:24.900742 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.900548 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3285330-1d1e-4d21-a032-df1a01f22bfc-oauth-serving-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:52:24.900742 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:24.900557 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3285330-1d1e-4d21-a032-df1a01f22bfc-console-serving-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:52:25.515330 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:25.515289 2564 generic.go:356] "Generic (PLEG): container finished" podID="c3285330-1d1e-4d21-a032-df1a01f22bfc" containerID="6943704e15646b4fc1909625d437ba26f30b2689c69df75971d7c89e84e0b227" exitCode=0 Jul 08 22:52:25.515330 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:25.515335 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd57b4cf4-9vhsl" Jul 08 22:52:25.515850 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:25.515373 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd57b4cf4-9vhsl" event={"ID":"c3285330-1d1e-4d21-a032-df1a01f22bfc","Type":"ContainerDied","Data":"6943704e15646b4fc1909625d437ba26f30b2689c69df75971d7c89e84e0b227"} Jul 08 22:52:25.515850 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:25.515435 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd57b4cf4-9vhsl" event={"ID":"c3285330-1d1e-4d21-a032-df1a01f22bfc","Type":"ContainerDied","Data":"932c08edcbfb2474417505791f0560bfd1a2eb4f15fe9e80e243d81cbb0f899f"} Jul 08 22:52:25.515850 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:25.515456 2564 scope.go:117] "RemoveContainer" containerID="6943704e15646b4fc1909625d437ba26f30b2689c69df75971d7c89e84e0b227" Jul 08 22:52:25.524221 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:25.524204 2564 scope.go:117] "RemoveContainer" containerID="6943704e15646b4fc1909625d437ba26f30b2689c69df75971d7c89e84e0b227" Jul 08 22:52:25.524479 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:52:25.524459 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6943704e15646b4fc1909625d437ba26f30b2689c69df75971d7c89e84e0b227\": container with ID starting with 6943704e15646b4fc1909625d437ba26f30b2689c69df75971d7c89e84e0b227 not found: ID does not exist" containerID="6943704e15646b4fc1909625d437ba26f30b2689c69df75971d7c89e84e0b227" Jul 08 22:52:25.524530 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:25.524488 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6943704e15646b4fc1909625d437ba26f30b2689c69df75971d7c89e84e0b227"} err="failed to get container status \"6943704e15646b4fc1909625d437ba26f30b2689c69df75971d7c89e84e0b227\": rpc error: code = NotFound desc = could not find container \"6943704e15646b4fc1909625d437ba26f30b2689c69df75971d7c89e84e0b227\": container with ID starting with 6943704e15646b4fc1909625d437ba26f30b2689c69df75971d7c89e84e0b227 not found: ID does not exist" Jul 08 22:52:25.535444 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:25.535419 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dd57b4cf4-9vhsl"] Jul 08 22:52:25.540071 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:25.540050 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dd57b4cf4-9vhsl"] Jul 08 22:52:26.265876 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:26.265836 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3285330-1d1e-4d21-a032-df1a01f22bfc" path="/var/lib/kubelet/pods/c3285330-1d1e-4d21-a032-df1a01f22bfc/volumes" Jul 08 22:52:27.347745 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:27.347718 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:52:27.355765 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:27.352459 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-587895c54b-tfndr" Jul 08 22:52:32.545954 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:32.545921 2564 generic.go:356] "Generic (PLEG): container finished" podID="1e961afb-3fe2-4fad-9c78-45b9370b073d" containerID="1ca96a6fe34604479f862c10b5ab0c239072c9fd5303d486cf469f2450e6831c" exitCode=0 Jul 08 22:52:32.546422 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:32.545993 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" event={"ID":"1e961afb-3fe2-4fad-9c78-45b9370b073d","Type":"ContainerDied","Data":"1ca96a6fe34604479f862c10b5ab0c239072c9fd5303d486cf469f2450e6831c"} Jul 08 22:52:32.546422 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:32.546329 2564 scope.go:117] "RemoveContainer" containerID="1ca96a6fe34604479f862c10b5ab0c239072c9fd5303d486cf469f2450e6831c" Jul 08 22:52:33.553015 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:33.552980 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xgdbp" event={"ID":"1e961afb-3fe2-4fad-9c78-45b9370b073d","Type":"ContainerStarted","Data":"5ecf05a85a37fc32e7e6123a4e2a3e9f1200cd027c8c292ddca1110bac7ed739"} Jul 08 22:52:42.867804 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:42.867724 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:52:42.965342 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:42.965307 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:52:49.149389 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:49.149352 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:52:49.168267 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:49.168240 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:52:49.634998 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:49.634968 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:52:58.723730 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:58.723693 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:52:58.776783 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:52:58.776750 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:00.099684 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:00.099647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:53:00.102140 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:00.102118 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed6eae7-7b43-47ba-b489-136fab2a8ade-metrics-certs\") pod \"network-metrics-daemon-rldqz\" (UID: \"4ed6eae7-7b43-47ba-b489-136fab2a8ade\") " pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:53:00.264983 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:00.264955 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-48cc2\"" Jul 08 22:53:00.272998 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:00.272978 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rldqz" Jul 08 22:53:00.399754 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:00.399730 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rldqz"] Jul 08 22:53:00.402596 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:53:00.402563 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed6eae7_7b43_47ba_b489_136fab2a8ade.slice/crio-ae7766b175d0731601df61556478d918a86f9aab2b392f3cb55be1e3dd5fd7d0 WatchSource:0}: Error finding container ae7766b175d0731601df61556478d918a86f9aab2b392f3cb55be1e3dd5fd7d0: Status 404 returned error can't find the container with id ae7766b175d0731601df61556478d918a86f9aab2b392f3cb55be1e3dd5fd7d0 Jul 08 22:53:00.666219 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:00.666143 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rldqz" event={"ID":"4ed6eae7-7b43-47ba-b489-136fab2a8ade","Type":"ContainerStarted","Data":"ae7766b175d0731601df61556478d918a86f9aab2b392f3cb55be1e3dd5fd7d0"} Jul 08 22:53:01.671694 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:01.671612 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rldqz" event={"ID":"4ed6eae7-7b43-47ba-b489-136fab2a8ade","Type":"ContainerStarted","Data":"f543605b3e70afc839f9d5f0b8bb755b73ec712a7e4819a16fdb63090886d932"} Jul 08 22:53:01.672042 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:01.671699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rldqz" event={"ID":"4ed6eae7-7b43-47ba-b489-136fab2a8ade","Type":"ContainerStarted","Data":"cd69a507d8097074d3e5aabd2f3887280fb98d557868e175a1c5ed566c6b6d20"} Jul 08 22:53:01.688438 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:01.688385 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rldqz" podStartSLOduration=374.735788142 podStartE2EDuration="6m15.688370464s" totalImagesPullingTime="952.582322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:53:01.685876783 +0000 UTC m=+375.941324487" watchObservedRunningTime="2026-07-08 22:53:01.688370464 +0000 UTC m=+375.943818212" Jul 08 22:53:01.759508 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:01.759472 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:01.814025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:01.814004 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:07.168546 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.168507 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 22:53:07.169044 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.168892 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="prometheus" containerID="cri-o://43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c" gracePeriod=600 Jul 08 22:53:07.169044 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.168900 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed" gracePeriod=600 Jul 08 22:53:07.169044 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.168930 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="kube-rbac-proxy-web" containerID="cri-o://9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d" gracePeriod=600 Jul 08 22:53:07.169044 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.168952 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="thanos-sidecar" containerID="cri-o://15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56" gracePeriod=600 Jul 08 22:53:07.169044 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.168970 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="kube-rbac-proxy" containerID="cri-o://bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3" gracePeriod=600 Jul 08 22:53:07.169326 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.169051 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="config-reloader" containerID="cri-o://e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8" gracePeriod=600 Jul 08 22:53:07.407007 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.406981 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.460395 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460316 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-0\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460395 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460348 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-tls\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460395 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460367 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-web-config\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460654 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460458 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-config-out\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460654 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460494 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-serving-certs-ca-bundle\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460654 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460537 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-2\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460654 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460568 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-metrics-client-certs\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460654 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460594 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-db\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460654 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460621 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-trusted-ca-bundle\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460932 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460659 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42468\" (UniqueName: \"kubernetes.io/projected/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-kube-api-access-42468\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460932 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460697 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-config\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460932 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460731 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-tls-assets\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460932 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460768 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460932 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460801 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-metrics-client-ca\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460932 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460828 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-grpc-tls\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460932 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460859 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-kubelet-serving-ca-bundle\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460932 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460901 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-1\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.460932 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460930 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.461449 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460963 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-kube-rbac-proxy\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.461449 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.460997 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-thanos-prometheus-http-client-file\") pod \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\" (UID: \"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe\") " Jul 08 22:53:07.461449 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.461021 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 22:53:07.461449 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.461281 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.461449 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.461328 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 22:53:07.462025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.461718 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 22:53:07.462025 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.461992 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 22:53:07.462571 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.462044 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 22:53:07.462982 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.462952 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 22:53:07.463124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.463071 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 22:53:07.463533 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.463371 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 22:53:07.463968 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.463941 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 22:53:07.464279 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.464244 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 22:53:07.464560 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.464530 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 22:53:07.464863 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.464838 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-config" (OuterVolumeSpecName: "config") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 22:53:07.465196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.465171 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-config-out" (OuterVolumeSpecName: "config-out") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 22:53:07.465621 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.465597 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 22:53:07.465927 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.465906 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 22:53:07.466358 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.466319 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 22:53:07.466560 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.466541 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 22:53:07.466909 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.466874 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-kube-api-access-42468" (OuterVolumeSpecName: "kube-api-access-42468") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "kube-api-access-42468". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 22:53:07.467019 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.466995 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 22:53:07.475404 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.475381 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-web-config" (OuterVolumeSpecName: "web-config") pod "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" (UID: "fffb1519-4d3d-4c5d-9149-32c6b0d49fbe"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 22:53:07.561963 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.561934 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-tls-assets\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.561963 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.561963 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.561979 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-metrics-client-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.561993 2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-grpc-tls\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562008 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562021 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562035 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562048 2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-kube-rbac-proxy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562062 2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562074 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562118 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562132 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-web-config\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562145 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-config-out\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562158 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562172 2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-secret-metrics-client-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562186 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-k8s-db\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562196 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562200 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562679 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562214 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42468\" (UniqueName: \"kubernetes.io/projected/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-kube-api-access-42468\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.562679 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.562229 2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe-config\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:53:07.697849 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.697818 2564 generic.go:356] "Generic (PLEG): container finished" podID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerID="4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed" exitCode=0 Jul 08 22:53:07.697849 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.697843 2564 generic.go:356] "Generic (PLEG): container finished" podID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerID="bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3" exitCode=0 Jul 08 22:53:07.697849 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.697850 2564 generic.go:356] "Generic (PLEG): container finished" podID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerID="9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d" exitCode=0 Jul 08 22:53:07.697849 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.697856 2564 generic.go:356] "Generic (PLEG): container finished" podID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerID="15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56" exitCode=0 Jul 08 22:53:07.697849 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.697861 2564 generic.go:356] "Generic (PLEG): container finished" podID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerID="e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8" exitCode=0 Jul 08 22:53:07.698176 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.697866 2564 generic.go:356] "Generic (PLEG): container finished" podID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerID="43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c" exitCode=0 Jul 08 22:53:07.698176 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.697889 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.698176 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.697904 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerDied","Data":"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed"} Jul 08 22:53:07.698176 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.697973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerDied","Data":"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3"} Jul 08 22:53:07.698176 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.698004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerDied","Data":"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d"} Jul 08 22:53:07.698176 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.698029 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerDied","Data":"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56"} Jul 08 22:53:07.698176 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.698037 2564 scope.go:117] "RemoveContainer" containerID="4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed" Jul 08 22:53:07.698176 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.698052 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerDied","Data":"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8"} Jul 08 22:53:07.698176 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.698077 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerDied","Data":"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c"} Jul 08 22:53:07.698176 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.698118 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fffb1519-4d3d-4c5d-9149-32c6b0d49fbe","Type":"ContainerDied","Data":"5b0c8533a00967eaf10c837ca6fbd9a19ffe35458a9a6c87983ca4d25b34a6cf"} Jul 08 22:53:07.706239 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.706222 2564 scope.go:117] "RemoveContainer" containerID="bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3" Jul 08 22:53:07.713156 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.713138 2564 scope.go:117] "RemoveContainer" containerID="9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d" Jul 08 22:53:07.721564 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.721543 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 22:53:07.722672 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.722582 2564 scope.go:117] "RemoveContainer" containerID="15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56" Jul 08 22:53:07.725490 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.725469 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 22:53:07.730519 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.730503 2564 scope.go:117] "RemoveContainer" containerID="e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8" Jul 08 22:53:07.737058 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.737030 2564 scope.go:117] "RemoveContainer" containerID="43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c" Jul 08 22:53:07.744358 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.744341 2564 scope.go:117] "RemoveContainer" containerID="6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf" Jul 08 22:53:07.750487 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750466 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 22:53:07.750760 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750749 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="init-config-reloader" Jul 08 22:53:07.750814 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750762 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="init-config-reloader" Jul 08 22:53:07.750814 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750775 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="config-reloader" Jul 08 22:53:07.750814 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750782 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="config-reloader" Jul 08 22:53:07.750814 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750789 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="kube-rbac-proxy-web" Jul 08 22:53:07.750814 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750795 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="kube-rbac-proxy-web" Jul 08 22:53:07.750814 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750801 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="kube-rbac-proxy" Jul 08 22:53:07.750814 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750806 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="kube-rbac-proxy" Jul 08 22:53:07.750814 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750811 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="kube-rbac-proxy-thanos" Jul 08 22:53:07.750814 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750816 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="kube-rbac-proxy-thanos" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750827 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="prometheus" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750832 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="prometheus" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750838 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82dddf27-89f2-45f8-ba41-74a7be616df4" containerName="registry" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750843 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="82dddf27-89f2-45f8-ba41-74a7be616df4" containerName="registry" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750848 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3285330-1d1e-4d21-a032-df1a01f22bfc" containerName="console" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750853 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3285330-1d1e-4d21-a032-df1a01f22bfc" containerName="console" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750867 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="thanos-sidecar" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750874 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="thanos-sidecar" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750919 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3285330-1d1e-4d21-a032-df1a01f22bfc" containerName="console" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750925 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="config-reloader" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750930 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="thanos-sidecar" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750934 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="kube-rbac-proxy" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750941 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="prometheus" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750945 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="kube-rbac-proxy-thanos" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750950 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="82dddf27-89f2-45f8-ba41-74a7be616df4" containerName="registry" Jul 08 22:53:07.751124 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.750957 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" containerName="kube-rbac-proxy-web" Jul 08 22:53:07.751766 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.751218 2564 scope.go:117] "RemoveContainer" containerID="4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed" Jul 08 22:53:07.751766 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:53:07.751499 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed\": container with ID starting with 4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed not found: ID does not exist" containerID="4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed" Jul 08 22:53:07.751766 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.751521 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed"} err="failed to get container status \"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed\": rpc error: code = NotFound desc = could not find container \"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed\": container with ID starting with 4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed not found: ID does not exist" Jul 08 22:53:07.751766 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.751540 2564 scope.go:117] "RemoveContainer" containerID="bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3" Jul 08 22:53:07.751920 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:53:07.751773 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3\": container with ID starting with bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3 not found: ID does not exist" containerID="bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3" Jul 08 22:53:07.751920 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.751791 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3"} err="failed to get container status \"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3\": rpc error: code = NotFound desc = could not find container \"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3\": container with ID starting with bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3 not found: ID does not exist" Jul 08 22:53:07.751920 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.751805 2564 scope.go:117] "RemoveContainer" containerID="9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d" Jul 08 22:53:07.752071 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:53:07.752050 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d\": container with ID starting with 9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d not found: ID does not exist" containerID="9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d" Jul 08 22:53:07.752130 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.752093 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d"} err="failed to get container status \"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d\": rpc error: code = NotFound desc = could not find container \"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d\": container with ID starting with 9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d not found: ID does not exist" Jul 08 22:53:07.752130 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.752114 2564 scope.go:117] "RemoveContainer" containerID="15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56" Jul 08 22:53:07.752389 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:53:07.752367 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56\": container with ID starting with 15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56 not found: ID does not exist" containerID="15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56" Jul 08 22:53:07.752456 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.752397 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56"} err="failed to get container status \"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56\": rpc error: code = NotFound desc = could not find container \"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56\": container with ID starting with 15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56 not found: ID does not exist" Jul 08 22:53:07.752456 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.752417 2564 scope.go:117] "RemoveContainer" containerID="e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8" Jul 08 22:53:07.752655 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:53:07.752639 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8\": container with ID starting with e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8 not found: ID does not exist" containerID="e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8" Jul 08 22:53:07.752700 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.752659 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8"} err="failed to get container status \"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8\": rpc error: code = NotFound desc = could not find container \"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8\": container with ID starting with e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8 not found: ID does not exist" Jul 08 22:53:07.752700 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.752673 2564 scope.go:117] "RemoveContainer" containerID="43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c" Jul 08 22:53:07.752905 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:53:07.752883 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c\": container with ID starting with 43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c not found: ID does not exist" containerID="43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c" Jul 08 22:53:07.752954 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.752910 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c"} err="failed to get container status \"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c\": rpc error: code = NotFound desc = could not find container \"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c\": container with ID starting with 43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c not found: ID does not exist" Jul 08 22:53:07.752954 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.752928 2564 scope.go:117] "RemoveContainer" containerID="6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf" Jul 08 22:53:07.753192 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:53:07.753173 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf\": container with ID starting with 6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf not found: ID does not exist" containerID="6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf" Jul 08 22:53:07.753252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.753197 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf"} err="failed to get container status \"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf\": rpc error: code = NotFound desc = could not find container \"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf\": container with ID starting with 6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf not found: ID does not exist" Jul 08 22:53:07.753252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.753218 2564 scope.go:117] "RemoveContainer" containerID="4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed" Jul 08 22:53:07.753437 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.753420 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed"} err="failed to get container status \"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed\": rpc error: code = NotFound desc = could not find container \"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed\": container with ID starting with 4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed not found: ID does not exist" Jul 08 22:53:07.753475 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.753437 2564 scope.go:117] "RemoveContainer" containerID="bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3" Jul 08 22:53:07.753653 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.753633 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3"} err="failed to get container status \"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3\": rpc error: code = NotFound desc = could not find container \"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3\": container with ID starting with bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3 not found: ID does not exist" Jul 08 22:53:07.753694 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.753654 2564 scope.go:117] "RemoveContainer" containerID="9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d" Jul 08 22:53:07.753901 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.753886 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d"} err="failed to get container status \"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d\": rpc error: code = NotFound desc = could not find container \"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d\": container with ID starting with 9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d not found: ID does not exist" Jul 08 22:53:07.753946 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.753901 2564 scope.go:117] "RemoveContainer" containerID="15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56" Jul 08 22:53:07.754138 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.754112 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56"} err="failed to get container status \"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56\": rpc error: code = NotFound desc = could not find container \"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56\": container with ID starting with 15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56 not found: ID does not exist" Jul 08 22:53:07.754138 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.754136 2564 scope.go:117] "RemoveContainer" containerID="e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8" Jul 08 22:53:07.754498 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.754474 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8"} err="failed to get container status \"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8\": rpc error: code = NotFound desc = could not find container \"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8\": container with ID starting with e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8 not found: ID does not exist" Jul 08 22:53:07.754498 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.754496 2564 scope.go:117] "RemoveContainer" containerID="43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c" Jul 08 22:53:07.754746 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.754729 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c"} err="failed to get container status \"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c\": rpc error: code = NotFound desc = could not find container \"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c\": container with ID starting with 43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c not found: ID does not exist" Jul 08 22:53:07.754801 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.754747 2564 scope.go:117] "RemoveContainer" containerID="6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf" Jul 08 22:53:07.754967 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.754947 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf"} err="failed to get container status \"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf\": rpc error: code = NotFound desc = could not find container \"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf\": container with ID starting with 6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf not found: ID does not exist" Jul 08 22:53:07.755009 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.754969 2564 scope.go:117] "RemoveContainer" containerID="4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed" Jul 08 22:53:07.755064 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.755049 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.755177 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.755156 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed"} err="failed to get container status \"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed\": rpc error: code = NotFound desc = could not find container \"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed\": container with ID starting with 4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed not found: ID does not exist" Jul 08 22:53:07.755228 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.755178 2564 scope.go:117] "RemoveContainer" containerID="bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3" Jul 08 22:53:07.755515 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.755493 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3"} err="failed to get container status \"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3\": rpc error: code = NotFound desc = could not find container \"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3\": container with ID starting with bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3 not found: ID does not exist" Jul 08 22:53:07.755599 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.755516 2564 scope.go:117] "RemoveContainer" containerID="9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d" Jul 08 22:53:07.755745 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.755718 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d"} err="failed to get container status \"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d\": rpc error: code = NotFound desc = could not find container \"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d\": container with ID starting with 9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d not found: ID does not exist" Jul 08 22:53:07.755852 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.755746 2564 scope.go:117] "RemoveContainer" containerID="15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56" Jul 08 22:53:07.756183 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.756158 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56"} err="failed to get container status \"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56\": rpc error: code = NotFound desc = could not find container \"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56\": container with ID starting with 15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56 not found: ID does not exist" Jul 08 22:53:07.756269 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.756186 2564 scope.go:117] "RemoveContainer" containerID="e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8" Jul 08 22:53:07.756441 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.756423 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8"} err="failed to get container status \"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8\": rpc error: code = NotFound desc = could not find container \"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8\": container with ID starting with e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8 not found: ID does not exist" Jul 08 22:53:07.756489 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.756442 2564 scope.go:117] "RemoveContainer" containerID="43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c" Jul 08 22:53:07.756669 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.756649 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c"} err="failed to get container status \"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c\": rpc error: code = NotFound desc = could not find container \"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c\": container with ID starting with 43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c not found: ID does not exist" Jul 08 22:53:07.756669 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.756668 2564 scope.go:117] "RemoveContainer" containerID="6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf" Jul 08 22:53:07.756921 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.756903 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf"} err="failed to get container status \"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf\": rpc error: code = NotFound desc = could not find container \"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf\": container with ID starting with 6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf not found: ID does not exist" Jul 08 22:53:07.756921 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.756921 2564 scope.go:117] "RemoveContainer" containerID="4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed" Jul 08 22:53:07.757151 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.757133 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed"} err="failed to get container status \"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed\": rpc error: code = NotFound desc = could not find container \"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed\": container with ID starting with 4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed not found: ID does not exist" Jul 08 22:53:07.757151 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.757148 2564 scope.go:117] "RemoveContainer" containerID="bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3" Jul 08 22:53:07.757370 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.757352 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3"} err="failed to get container status \"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3\": rpc error: code = NotFound desc = could not find container \"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3\": container with ID starting with bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3 not found: ID does not exist" Jul 08 22:53:07.757370 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.757369 2564 scope.go:117] "RemoveContainer" containerID="9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d" Jul 08 22:53:07.757621 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.757602 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d"} err="failed to get container status \"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d\": rpc error: code = NotFound desc = could not find container \"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d\": container with ID starting with 9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d not found: ID does not exist" Jul 08 22:53:07.757691 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.757623 2564 scope.go:117] "RemoveContainer" containerID="15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56" Jul 08 22:53:07.757744 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.757703 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 08 22:53:07.757744 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.757711 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 08 22:53:07.757907 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.757884 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56"} err="failed to get container status \"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56\": rpc error: code = NotFound desc = could not find container \"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56\": container with ID starting with 15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56 not found: ID does not exist" Jul 08 22:53:07.757983 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.757905 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 08 22:53:07.757983 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.757954 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 08 22:53:07.758108 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.757980 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-66664lm5btme8\"" Jul 08 22:53:07.758108 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.757911 2564 scope.go:117] "RemoveContainer" containerID="e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8" Jul 08 22:53:07.758212 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.758194 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nh89h\"" Jul 08 22:53:07.758291 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.758221 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 08 22:53:07.758291 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.758249 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 08 22:53:07.758390 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.758302 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 08 22:53:07.758390 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.758338 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 08 22:53:07.758390 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.758357 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 08 22:53:07.758528 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.758393 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 08 22:53:07.758528 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.758429 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 08 22:53:07.758528 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.758448 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 08 22:53:07.758671 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.758569 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8"} err="failed to get container status \"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8\": rpc error: code = NotFound desc = could not find container \"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8\": container with ID starting with e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8 not found: ID does not exist" Jul 08 22:53:07.758671 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.758591 2564 scope.go:117] "RemoveContainer" containerID="43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c" Jul 08 22:53:07.758857 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.758836 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c"} err="failed to get container status \"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c\": rpc error: code = NotFound desc = could not find container \"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c\": container with ID starting with 43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c not found: ID does not exist" Jul 08 22:53:07.758857 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.758857 2564 scope.go:117] "RemoveContainer" containerID="6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf" Jul 08 22:53:07.759170 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.759150 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf"} err="failed to get container status \"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf\": rpc error: code = NotFound desc = could not find container \"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf\": container with ID starting with 6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf not found: ID does not exist" Jul 08 22:53:07.759170 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.759170 2564 scope.go:117] "RemoveContainer" containerID="4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed" Jul 08 22:53:07.759396 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.759375 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed"} err="failed to get container status \"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed\": rpc error: code = NotFound desc = could not find container \"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed\": container with ID starting with 4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed not found: ID does not exist" Jul 08 22:53:07.759474 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.759397 2564 scope.go:117] "RemoveContainer" containerID="bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3" Jul 08 22:53:07.759672 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.759651 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3"} err="failed to get container status \"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3\": rpc error: code = NotFound desc = could not find container \"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3\": container with ID starting with bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3 not found: ID does not exist" Jul 08 22:53:07.759672 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.759670 2564 scope.go:117] "RemoveContainer" containerID="9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d" Jul 08 22:53:07.759936 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.759909 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d"} err="failed to get container status \"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d\": rpc error: code = NotFound desc = could not find container \"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d\": container with ID starting with 9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d not found: ID does not exist" Jul 08 22:53:07.759936 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.759934 2564 scope.go:117] "RemoveContainer" containerID="15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56" Jul 08 22:53:07.760187 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.760162 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56"} err="failed to get container status \"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56\": rpc error: code = NotFound desc = could not find container \"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56\": container with ID starting with 15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56 not found: ID does not exist" Jul 08 22:53:07.760258 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.760188 2564 scope.go:117] "RemoveContainer" containerID="e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8" Jul 08 22:53:07.760466 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.760446 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8"} err="failed to get container status \"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8\": rpc error: code = NotFound desc = could not find container \"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8\": container with ID starting with e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8 not found: ID does not exist" Jul 08 22:53:07.760466 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.760465 2564 scope.go:117] "RemoveContainer" containerID="43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c" Jul 08 22:53:07.760770 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.760718 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c"} err="failed to get container status \"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c\": rpc error: code = NotFound desc = could not find container \"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c\": container with ID starting with 43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c not found: ID does not exist" Jul 08 22:53:07.760770 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.760746 2564 scope.go:117] "RemoveContainer" containerID="6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf" Jul 08 22:53:07.761201 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.761067 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 08 22:53:07.761300 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.761212 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf"} err="failed to get container status \"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf\": rpc error: code = NotFound desc = could not find container \"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf\": container with ID starting with 6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf not found: ID does not exist" Jul 08 22:53:07.761300 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.761233 2564 scope.go:117] "RemoveContainer" containerID="4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed" Jul 08 22:53:07.761492 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.761472 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed"} err="failed to get container status \"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed\": rpc error: code = NotFound desc = could not find container \"4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed\": container with ID starting with 4a1c86f954ac77b219aa8ac9ec2f64ce18f806bafbc917e3522565011ac08aed not found: ID does not exist" Jul 08 22:53:07.761492 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.761493 2564 scope.go:117] "RemoveContainer" containerID="bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3" Jul 08 22:53:07.761709 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.761689 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3"} err="failed to get container status \"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3\": rpc error: code = NotFound desc = could not find container \"bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3\": container with ID starting with bb1ecf057eec631bdfc0a00b957a396139275939e444f059f5db31a1db12f3a3 not found: ID does not exist" Jul 08 22:53:07.761709 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.761709 2564 scope.go:117] "RemoveContainer" containerID="9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d" Jul 08 22:53:07.761910 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.761890 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d"} err="failed to get container status \"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d\": rpc error: code = NotFound desc = could not find container \"9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d\": container with ID starting with 9a26969489f952b91300a9f7d0fa511d6af421d0c0c152fa9f2c2622d9c7807d not found: ID does not exist" Jul 08 22:53:07.761970 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.761910 2564 scope.go:117] "RemoveContainer" containerID="15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56" Jul 08 22:53:07.762171 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.762150 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56"} err="failed to get container status \"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56\": rpc error: code = NotFound desc = could not find container \"15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56\": container with ID starting with 15091652765fe80b950865c2cdef6bac8fd20562f15fc0526ac580a8b43aee56 not found: ID does not exist" Jul 08 22:53:07.762256 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.762172 2564 scope.go:117] "RemoveContainer" containerID="e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8" Jul 08 22:53:07.762464 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.762428 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8"} err="failed to get container status \"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8\": rpc error: code = NotFound desc = could not find container \"e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8\": container with ID starting with e25dfd0f7ab816329f34357f47458a549d2081c38037d6d1b71f2d3415a9bca8 not found: ID does not exist" Jul 08 22:53:07.762464 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.762447 2564 scope.go:117] "RemoveContainer" containerID="43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c" Jul 08 22:53:07.762682 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.762663 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c"} err="failed to get container status \"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c\": rpc error: code = NotFound desc = could not find container \"43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c\": container with ID starting with 43b6e4bdab7ced28a5a5cdc9500a26acd9d557162e0110d97dcf1e1053cf5b6c not found: ID does not exist" Jul 08 22:53:07.762682 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.762681 2564 scope.go:117] "RemoveContainer" containerID="6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf" Jul 08 22:53:07.762926 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.762907 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf"} err="failed to get container status \"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf\": rpc error: code = NotFound desc = could not find container \"6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf\": container with ID starting with 6d46a0c5adc125fa664524cdc4a1524056fe823bfffded6b231745ab2174dcaf not found: ID does not exist" Jul 08 22:53:07.763038 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763022 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 08 22:53:07.763333 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763315 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.763420 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763364 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.763420 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.763420 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763403 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-config\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.763420 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a0b6fd7-dd32-4672-8c47-286e58651841-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.763612 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763442 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.763612 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763492 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.763612 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.763700 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a0b6fd7-dd32-4672-8c47-286e58651841-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.763700 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763640 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.763700 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.763804 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.763804 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.763804 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763761 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47prz\" (UniqueName: \"kubernetes.io/projected/8a0b6fd7-dd32-4672-8c47-286e58651841-kube-api-access-47prz\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.763804 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.764004 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.764004 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763867 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.764004 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763894 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.764004 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.763965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.764176 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.764005 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.775136 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.775111 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 22:53:07.864695 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.864665 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.864877 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.864704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.864877 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.864721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.864877 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.864745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.864877 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.864773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.864877 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.864806 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.864877 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.864851 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.864877 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.864872 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.864894 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-config\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.864919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a0b6fd7-dd32-4672-8c47-286e58651841-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.864940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.864964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.864991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.865027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a0b6fd7-dd32-4672-8c47-286e58651841-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.865056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.865112 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.865132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.865165 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.865192 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47prz\" (UniqueName: \"kubernetes.io/projected/8a0b6fd7-dd32-4672-8c47-286e58651841-kube-api-access-47prz\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.865211 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865252 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.865217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865974 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.865663 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.865974 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.865751 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.866096 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.866029 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.866096 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.866053 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.867670 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.866707 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.867792 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.867763 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.868936 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.868908 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a0b6fd7-dd32-4672-8c47-286e58651841-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.869048 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.869027 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.869449 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.869416 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a0b6fd7-dd32-4672-8c47-286e58651841-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.869600 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.869505 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.869871 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.869846 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.870128 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.870059 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.870237 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.870141 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a0b6fd7-dd32-4672-8c47-286e58651841-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.870237 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.870157 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.870237 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.870139 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.870410 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.870256 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.870410 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.870384 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-config\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.871316 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.871297 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a0b6fd7-dd32-4672-8c47-286e58651841-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:07.875496 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:07.875479 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47prz\" (UniqueName: \"kubernetes.io/projected/8a0b6fd7-dd32-4672-8c47-286e58651841-kube-api-access-47prz\") pod \"prometheus-k8s-0\" (UID: \"8a0b6fd7-dd32-4672-8c47-286e58651841\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:08.067122 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:08.067072 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:08.200347 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:08.200303 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 22:53:08.205043 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:53:08.205016 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0b6fd7_dd32_4672_8c47_286e58651841.slice/crio-96e066f70301c7e2cb23f8078394d837d55b5a587f8260be49c3932d9a00ac0d WatchSource:0}: Error finding container 96e066f70301c7e2cb23f8078394d837d55b5a587f8260be49c3932d9a00ac0d: Status 404 returned error can't find the container with id 96e066f70301c7e2cb23f8078394d837d55b5a587f8260be49c3932d9a00ac0d Jul 08 22:53:08.267247 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:08.267213 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fffb1519-4d3d-4c5d-9149-32c6b0d49fbe" path="/var/lib/kubelet/pods/fffb1519-4d3d-4c5d-9149-32c6b0d49fbe/volumes" Jul 08 22:53:08.703874 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:08.703838 2564 generic.go:356] "Generic (PLEG): container finished" podID="8a0b6fd7-dd32-4672-8c47-286e58651841" containerID="173b329c20106e5d237d5d2edc78d001d9f3d824013aab8fe4bf2368171767a3" exitCode=0 Jul 08 22:53:08.704077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:08.703922 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b6fd7-dd32-4672-8c47-286e58651841","Type":"ContainerDied","Data":"173b329c20106e5d237d5d2edc78d001d9f3d824013aab8fe4bf2368171767a3"} Jul 08 22:53:08.704077 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:08.704002 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b6fd7-dd32-4672-8c47-286e58651841","Type":"ContainerStarted","Data":"96e066f70301c7e2cb23f8078394d837d55b5a587f8260be49c3932d9a00ac0d"} Jul 08 22:53:09.712356 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:09.712323 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b6fd7-dd32-4672-8c47-286e58651841","Type":"ContainerStarted","Data":"d3b16f1437c5cec7deb8ec60a6a2e6c065d25abbfa0fd0fc2ccd43fefff5ff23"} Jul 08 22:53:09.712733 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:09.712386 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b6fd7-dd32-4672-8c47-286e58651841","Type":"ContainerStarted","Data":"8aed0e26a4fb5c2ff2e0b72d77196a11c54811d3dec99454a9b04c8589b4ab67"} Jul 08 22:53:09.712733 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:09.712415 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b6fd7-dd32-4672-8c47-286e58651841","Type":"ContainerStarted","Data":"0864291c0fcd8fd97fd96af0445c4bbd85e909b43964eb6947cd7195ec669d63"} Jul 08 22:53:09.712733 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:09.712442 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b6fd7-dd32-4672-8c47-286e58651841","Type":"ContainerStarted","Data":"4fdc753db03405c03dd99466f8c8847c49bcf900a7d71106c18fdaf3ad6d8311"} Jul 08 22:53:09.712733 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:09.712470 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b6fd7-dd32-4672-8c47-286e58651841","Type":"ContainerStarted","Data":"e41b6e56b034f5c21394d458a4ffb0c4e010e77ec1eea9a44d96a1aa156b05db"} Jul 08 22:53:09.712733 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:09.712494 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b6fd7-dd32-4672-8c47-286e58651841","Type":"ContainerStarted","Data":"49ee62f5535d071ceb08405c75bb9d6ce72d1251ffba4477c63008a0fb31f13d"} Jul 08 22:53:09.742230 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:09.742184 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.72438557 podStartE2EDuration="2.742166943s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.781373ms" isStatefulPod=false podCreationTimestamp="2026-07-08 22:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:53:09.739711325 +0000 UTC m=+383.995159025" watchObservedRunningTime="2026-07-08 22:53:09.742166943 +0000 UTC m=+383.997614646" Jul 08 22:53:10.567682 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:10.567638 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:10.616728 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:10.616694 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:13.067662 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:13.067624 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:53:15.264234 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:15.264173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:15.364562 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:15.364520 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:17.005102 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:17.005057 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:17.056353 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:17.056325 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:20.961809 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:20.961770 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:21.017168 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:21.017143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:21.067329 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:21.067307 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:23.920824 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:23.920787 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:23.972234 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:23.972200 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:28.019820 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:28.019771 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:28.120904 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:28.120870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:28.168283 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:28.168245 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:28.747994 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:28.747956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:43.125605 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:43.125550 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:53:43.225059 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:53:43.225027 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:08.067344 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:08.067256 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:54:08.082805 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:08.082783 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:54:08.526463 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:08.526435 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:08.976512 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:08.976445 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 22:54:09.736606 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:09.736569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:10.553621 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:10.553584 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:13.338747 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:13.338702 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:13.387487 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:13.387459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:14.440851 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:14.440818 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:14.545621 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:14.545589 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:15.369617 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:15.369574 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:15.471967 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:15.471937 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:17.175474 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:17.175443 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:17.754928 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:17.754878 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:20.135444 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:20.135400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:20.193463 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:20.193431 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:26.564097 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:26.564015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:26.746179 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:26.746145 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zvdk2"] Jul 08 22:54:26.748620 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:26.748601 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-zvdk2" Jul 08 22:54:26.751059 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:26.751039 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 08 22:54:26.751201 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:26.751185 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 08 22:54:26.752153 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:26.752135 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-x7sr4\"" Jul 08 22:54:26.758260 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:26.758239 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zvdk2"] Jul 08 22:54:26.813219 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:26.813187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e83dba1-a534-4e1b-8e58-5737c24d9a1b-bound-sa-token\") pod \"cert-manager-55b4d954d5-zvdk2\" (UID: \"6e83dba1-a534-4e1b-8e58-5737c24d9a1b\") " pod="cert-manager/cert-manager-55b4d954d5-zvdk2" Jul 08 22:54:26.813391 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:26.813244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw9xm\" (UniqueName: \"kubernetes.io/projected/6e83dba1-a534-4e1b-8e58-5737c24d9a1b-kube-api-access-pw9xm\") pod \"cert-manager-55b4d954d5-zvdk2\" (UID: \"6e83dba1-a534-4e1b-8e58-5737c24d9a1b\") " pod="cert-manager/cert-manager-55b4d954d5-zvdk2" Jul 08 22:54:26.914707 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:26.914630 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e83dba1-a534-4e1b-8e58-5737c24d9a1b-bound-sa-token\") pod \"cert-manager-55b4d954d5-zvdk2\" (UID: \"6e83dba1-a534-4e1b-8e58-5737c24d9a1b\") " pod="cert-manager/cert-manager-55b4d954d5-zvdk2" Jul 08 22:54:26.914707 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:26.914698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw9xm\" (UniqueName: \"kubernetes.io/projected/6e83dba1-a534-4e1b-8e58-5737c24d9a1b-kube-api-access-pw9xm\") pod \"cert-manager-55b4d954d5-zvdk2\" (UID: \"6e83dba1-a534-4e1b-8e58-5737c24d9a1b\") " pod="cert-manager/cert-manager-55b4d954d5-zvdk2" Jul 08 22:54:26.922394 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:26.922366 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e83dba1-a534-4e1b-8e58-5737c24d9a1b-bound-sa-token\") pod \"cert-manager-55b4d954d5-zvdk2\" (UID: \"6e83dba1-a534-4e1b-8e58-5737c24d9a1b\") " pod="cert-manager/cert-manager-55b4d954d5-zvdk2" Jul 08 22:54:26.922536 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:26.922496 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw9xm\" (UniqueName: \"kubernetes.io/projected/6e83dba1-a534-4e1b-8e58-5737c24d9a1b-kube-api-access-pw9xm\") pod \"cert-manager-55b4d954d5-zvdk2\" (UID: \"6e83dba1-a534-4e1b-8e58-5737c24d9a1b\") " pod="cert-manager/cert-manager-55b4d954d5-zvdk2" Jul 08 22:54:27.064931 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:27.064898 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-zvdk2" Jul 08 22:54:27.192453 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:27.192430 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zvdk2"] Jul 08 22:54:27.194435 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:54:27.194406 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e83dba1_a534_4e1b_8e58_5737c24d9a1b.slice/crio-ebd6905703aa11ad3affa9485b9b82e5f335b7ca64222be7cb1091dab031cab0 WatchSource:0}: Error finding container ebd6905703aa11ad3affa9485b9b82e5f335b7ca64222be7cb1091dab031cab0: Status 404 returned error can't find the container with id ebd6905703aa11ad3affa9485b9b82e5f335b7ca64222be7cb1091dab031cab0 Jul 08 22:54:27.748322 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:27.748283 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:27.801521 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:27.801480 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:28.038774 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:28.038658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-zvdk2" event={"ID":"6e83dba1-a534-4e1b-8e58-5737c24d9a1b","Type":"ContainerStarted","Data":"ebd6905703aa11ad3affa9485b9b82e5f335b7ca64222be7cb1091dab031cab0"} Jul 08 22:54:31.053367 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:31.053330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-zvdk2" event={"ID":"6e83dba1-a534-4e1b-8e58-5737c24d9a1b","Type":"ContainerStarted","Data":"810a1e0a55135dd5cfa33e2f8bb634e30b51fbc6cc910752650a961d4eeee1ce"} Jul 08 22:54:31.068695 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:31.068645 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-zvdk2" podStartSLOduration=1.958724988 podStartE2EDuration="5.068629504s" totalImagesPullingTime="3.109904516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:54:31.067564086 +0000 UTC m=+465.323011791" watchObservedRunningTime="2026-07-08 22:54:31.068629504 +0000 UTC m=+465.324077207" Jul 08 22:54:36.554575 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:36.554502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:36.600909 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:36.600841 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:37.663555 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:37.663511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:37.763150 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:37.763074 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:39.491442 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:39.491403 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5642z"] Jul 08 22:54:39.494627 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:39.494607 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5642z" Jul 08 22:54:39.497341 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:39.497321 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 22:54:39.497420 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:39.497395 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 22:54:39.498423 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:39.498406 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xj55t\"" Jul 08 22:54:39.501979 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:39.501950 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5642z"] Jul 08 22:54:39.622655 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:39.622612 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9615fd53-e65a-48ac-a76e-d769f2194c6e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5642z\" (UID: \"9615fd53-e65a-48ac-a76e-d769f2194c6e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5642z" Jul 08 22:54:39.622826 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:39.622674 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5qsv\" (UniqueName: \"kubernetes.io/projected/9615fd53-e65a-48ac-a76e-d769f2194c6e-kube-api-access-j5qsv\") pod \"openshift-lws-operator-fd8ccff4c-5642z\" (UID: \"9615fd53-e65a-48ac-a76e-d769f2194c6e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5642z" Jul 08 22:54:39.723387 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:39.723350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9615fd53-e65a-48ac-a76e-d769f2194c6e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5642z\" (UID: \"9615fd53-e65a-48ac-a76e-d769f2194c6e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5642z" Jul 08 22:54:39.723544 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:39.723412 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5qsv\" (UniqueName: \"kubernetes.io/projected/9615fd53-e65a-48ac-a76e-d769f2194c6e-kube-api-access-j5qsv\") pod \"openshift-lws-operator-fd8ccff4c-5642z\" (UID: \"9615fd53-e65a-48ac-a76e-d769f2194c6e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5642z" Jul 08 22:54:39.723711 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:39.723693 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9615fd53-e65a-48ac-a76e-d769f2194c6e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5642z\" (UID: \"9615fd53-e65a-48ac-a76e-d769f2194c6e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5642z" Jul 08 22:54:39.732716 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:39.732694 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5qsv\" (UniqueName: \"kubernetes.io/projected/9615fd53-e65a-48ac-a76e-d769f2194c6e-kube-api-access-j5qsv\") pod \"openshift-lws-operator-fd8ccff4c-5642z\" (UID: \"9615fd53-e65a-48ac-a76e-d769f2194c6e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5642z" Jul 08 22:54:39.804419 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:39.804384 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5642z" Jul 08 22:54:39.938702 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:39.938678 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5642z"] Jul 08 22:54:39.940685 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:54:39.940657 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9615fd53_e65a_48ac_a76e_d769f2194c6e.slice/crio-e6e0fa5476d4464671c21187762c99d97d2ea6beaa2ef274f2422fef10df061b WatchSource:0}: Error finding container e6e0fa5476d4464671c21187762c99d97d2ea6beaa2ef274f2422fef10df061b: Status 404 returned error can't find the container with id e6e0fa5476d4464671c21187762c99d97d2ea6beaa2ef274f2422fef10df061b Jul 08 22:54:40.091311 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:40.091224 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5642z" event={"ID":"9615fd53-e65a-48ac-a76e-d769f2194c6e","Type":"ContainerStarted","Data":"e6e0fa5476d4464671c21187762c99d97d2ea6beaa2ef274f2422fef10df061b"} Jul 08 22:54:43.107127 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:43.107068 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5642z" event={"ID":"9615fd53-e65a-48ac-a76e-d769f2194c6e","Type":"ContainerStarted","Data":"3123d08267047bb50bd55f0779df29bcb2003cf51e1b5d97d675ce7606500892"} Jul 08 22:54:43.122801 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:43.122751 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5642z" podStartSLOduration=1.051348116 podStartE2EDuration="4.122736328s" totalImagesPullingTime="3.071388212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:54:43.121761671 +0000 UTC m=+477.377209374" watchObservedRunningTime="2026-07-08 22:54:43.122736328 +0000 UTC m=+477.378184031" Jul 08 22:54:43.787162 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:43.787128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:43.836503 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:43.836478 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:46.395493 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:46.395458 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:58.700058 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.700016 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:58.751915 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.751888 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj"] Jul 08 22:54:58.755135 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.755119 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:54:58.757614 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.757590 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 08 22:54:58.758582 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.758566 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 08 22:54:58.758673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.758581 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 08 22:54:58.758673 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.758637 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-88gxl\"" Jul 08 22:54:58.765043 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.765025 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj"] Jul 08 22:54:58.771460 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.771440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5a0819b-b6e8-40cf-ab85-76c8991b268e-metrics-cert\") pod \"lws-controller-manager-6cfd684dd8-dllmj\" (UID: \"e5a0819b-b6e8-40cf-ab85-76c8991b268e\") " pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:54:58.771544 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.771474 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jktzr\" (UniqueName: \"kubernetes.io/projected/e5a0819b-b6e8-40cf-ab85-76c8991b268e-kube-api-access-jktzr\") pod \"lws-controller-manager-6cfd684dd8-dllmj\" (UID: \"e5a0819b-b6e8-40cf-ab85-76c8991b268e\") " pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:54:58.771544 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.771495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5a0819b-b6e8-40cf-ab85-76c8991b268e-cert\") pod \"lws-controller-manager-6cfd684dd8-dllmj\" (UID: \"e5a0819b-b6e8-40cf-ab85-76c8991b268e\") " pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:54:58.771639 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.771613 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e5a0819b-b6e8-40cf-ab85-76c8991b268e-manager-config\") pod \"lws-controller-manager-6cfd684dd8-dllmj\" (UID: \"e5a0819b-b6e8-40cf-ab85-76c8991b268e\") " pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:54:58.779654 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.779633 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:54:58.872483 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.872456 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e5a0819b-b6e8-40cf-ab85-76c8991b268e-manager-config\") pod \"lws-controller-manager-6cfd684dd8-dllmj\" (UID: \"e5a0819b-b6e8-40cf-ab85-76c8991b268e\") " pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:54:58.872626 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.872520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5a0819b-b6e8-40cf-ab85-76c8991b268e-metrics-cert\") pod \"lws-controller-manager-6cfd684dd8-dllmj\" (UID: \"e5a0819b-b6e8-40cf-ab85-76c8991b268e\") " pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:54:58.872626 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.872548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jktzr\" (UniqueName: \"kubernetes.io/projected/e5a0819b-b6e8-40cf-ab85-76c8991b268e-kube-api-access-jktzr\") pod \"lws-controller-manager-6cfd684dd8-dllmj\" (UID: \"e5a0819b-b6e8-40cf-ab85-76c8991b268e\") " pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:54:58.872626 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.872571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5a0819b-b6e8-40cf-ab85-76c8991b268e-cert\") pod \"lws-controller-manager-6cfd684dd8-dllmj\" (UID: \"e5a0819b-b6e8-40cf-ab85-76c8991b268e\") " pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:54:58.873306 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.873284 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e5a0819b-b6e8-40cf-ab85-76c8991b268e-manager-config\") pod \"lws-controller-manager-6cfd684dd8-dllmj\" (UID: \"e5a0819b-b6e8-40cf-ab85-76c8991b268e\") " pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:54:58.875005 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.874985 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5a0819b-b6e8-40cf-ab85-76c8991b268e-cert\") pod \"lws-controller-manager-6cfd684dd8-dllmj\" (UID: \"e5a0819b-b6e8-40cf-ab85-76c8991b268e\") " pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:54:58.875128 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.875019 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5a0819b-b6e8-40cf-ab85-76c8991b268e-metrics-cert\") pod \"lws-controller-manager-6cfd684dd8-dllmj\" (UID: \"e5a0819b-b6e8-40cf-ab85-76c8991b268e\") " pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:54:58.880760 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:58.880741 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jktzr\" (UniqueName: \"kubernetes.io/projected/e5a0819b-b6e8-40cf-ab85-76c8991b268e-kube-api-access-jktzr\") pod \"lws-controller-manager-6cfd684dd8-dllmj\" (UID: \"e5a0819b-b6e8-40cf-ab85-76c8991b268e\") " pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:54:59.064824 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:59.064802 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:54:59.197429 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:54:59.197407 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj"] Jul 08 22:54:59.199978 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:54:59.199949 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a0819b_b6e8_40cf_ab85_76c8991b268e.slice/crio-38740d2cf2a966b61b135189974901295891509f4e3296f9661866133804296b WatchSource:0}: Error finding container 38740d2cf2a966b61b135189974901295891509f4e3296f9661866133804296b: Status 404 returned error can't find the container with id 38740d2cf2a966b61b135189974901295891509f4e3296f9661866133804296b Jul 08 22:55:00.185415 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:00.185377 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" event={"ID":"e5a0819b-b6e8-40cf-ab85-76c8991b268e","Type":"ContainerStarted","Data":"38740d2cf2a966b61b135189974901295891509f4e3296f9661866133804296b"} Jul 08 22:55:01.190957 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:01.190877 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" event={"ID":"e5a0819b-b6e8-40cf-ab85-76c8991b268e","Type":"ContainerStarted","Data":"a171d094e6dcae3cfe486e3f72a861627020f674549d3c74ce0036dd759fcfbc"} Jul 08 22:55:01.191316 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:01.190995 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:55:01.209461 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:01.209411 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" podStartSLOduration=1.647684631 podStartE2EDuration="3.209397133s" totalImagesPullingTime="1.561712502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:55:01.208428557 +0000 UTC m=+495.463876261" watchObservedRunningTime="2026-07-08 22:55:01.209397133 +0000 UTC m=+495.464844837" Jul 08 22:55:01.426522 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:01.426442 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:01.517799 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:01.517768 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:02.034858 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.034825 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:02.235101 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.235061 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:02.339707 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.339629 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:02.412223 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.412187 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj"] Jul 08 22:55:02.415574 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.415553 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.422433 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.422405 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 22:55:02.422571 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.422440 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 22:55:02.422571 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.422458 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 22:55:02.422571 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.422495 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 22:55:02.422571 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.422418 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zbz6j\"" Jul 08 22:55:02.450622 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.450597 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj"] Jul 08 22:55:02.495836 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.495806 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-manifests\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.495962 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.495847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.495962 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.495882 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-tmp\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.495962 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.495927 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.496138 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.495981 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp2wj\" (UniqueName: \"kubernetes.io/projected/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-kube-api-access-sp2wj\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.540041 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.540015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:02.596403 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.596335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp2wj\" (UniqueName: \"kubernetes.io/projected/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-kube-api-access-sp2wj\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.596403 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.596376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-manifests\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.596636 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.596411 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.596636 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.596446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-tmp\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.596636 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.596487 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.596836 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.596812 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-manifests\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.598719 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.598694 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-tmp\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.598972 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.598951 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.599134 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.599117 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.603785 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.603759 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp2wj\" (UniqueName: \"kubernetes.io/projected/1e774b31-f7ae-404c-a1d0-93f89b2c00ab-kube-api-access-sp2wj\") pod \"opendatahub-operator-controller-manager-86d776fd96-jtkxj\" (UID: \"1e774b31-f7ae-404c-a1d0-93f89b2c00ab\") " pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.641188 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.641162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:02.732766 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.732737 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:02.740301 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.740265 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:02.805905 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.805879 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:02.867867 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.867796 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj"] Jul 08 22:55:02.870902 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:55:02.870872 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e774b31_f7ae_404c_a1d0_93f89b2c00ab.slice/crio-25945f9be31d8569b4da80a507c78840e9be3fced8f9fee4c471410f729b09b5 WatchSource:0}: Error finding container 25945f9be31d8569b4da80a507c78840e9be3fced8f9fee4c471410f729b09b5: Status 404 returned error can't find the container with id 25945f9be31d8569b4da80a507c78840e9be3fced8f9fee4c471410f729b09b5 Jul 08 22:55:02.877074 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.877052 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:02.936418 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:02.936390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:03.034365 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:03.034336 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:03.201901 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:03.201822 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" event={"ID":"1e774b31-f7ae-404c-a1d0-93f89b2c00ab","Type":"ContainerStarted","Data":"25945f9be31d8569b4da80a507c78840e9be3fced8f9fee4c471410f729b09b5"} Jul 08 22:55:06.217326 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:06.217255 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" event={"ID":"1e774b31-f7ae-404c-a1d0-93f89b2c00ab","Type":"ContainerStarted","Data":"d2228357ac9565cbb2fdfb139226553ec6e5b875bd8ed3c71825ef9b6774daf7"} Jul 08 22:55:07.223482 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:07.223406 2564 generic.go:356] "Generic (PLEG): container finished" podID="1e774b31-f7ae-404c-a1d0-93f89b2c00ab" containerID="d2228357ac9565cbb2fdfb139226553ec6e5b875bd8ed3c71825ef9b6774daf7" exitCode=0 Jul 08 22:55:07.223482 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:07.223473 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" event={"ID":"1e774b31-f7ae-404c-a1d0-93f89b2c00ab","Type":"ContainerDied","Data":"d2228357ac9565cbb2fdfb139226553ec6e5b875bd8ed3c71825ef9b6774daf7"} Jul 08 22:55:08.229545 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:08.229496 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" event={"ID":"1e774b31-f7ae-404c-a1d0-93f89b2c00ab","Type":"ContainerStarted","Data":"14c86844cbeabb1266d2c6d24b22a1c3741ccc8f07336653c57dd49fe8619f93"} Jul 08 22:55:08.229992 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:08.229620 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:08.252426 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:08.252375 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" podStartSLOduration=1.8990335520000001 podStartE2EDuration="6.25236014s" totalImagesPullingTime="3.326138196s" totalInitContainerRuntime="1.027188392s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:55:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:55:08.250073786 +0000 UTC m=+502.505521488" watchObservedRunningTime="2026-07-08 22:55:08.25236014 +0000 UTC m=+502.507807842" Jul 08 22:55:12.197789 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:12.197762 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cfd684dd8-dllmj" Jul 08 22:55:14.877012 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:14.876980 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:14.977730 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:14.977704 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:15.396679 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:15.396647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:18.714515 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:18.714481 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:18.771411 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:18.771387 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:19.236227 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:19.236201 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-86d776fd96-jtkxj" Jul 08 22:55:21.730117 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:21.730070 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:21.832558 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:21.832514 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:22.073134 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.073065 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:22.207461 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.207433 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78df955549-w6fpf"] Jul 08 22:55:22.214347 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.214323 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78df955549-w6fpf" Jul 08 22:55:22.217237 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.217210 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 22:55:22.218071 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.218052 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 22:55:22.218071 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.218061 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 22:55:22.218071 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.218057 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 22:55:22.218313 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.218076 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2fh2j\"" Jul 08 22:55:22.227129 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.227104 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78df955549-w6fpf"] Jul 08 22:55:22.265982 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.265957 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/361c10e9-fc5c-4f69-9647-ae45cebfe624-tmp\") pod \"kube-auth-proxy-78df955549-w6fpf\" (UID: \"361c10e9-fc5c-4f69-9647-ae45cebfe624\") " pod="openshift-ingress/kube-auth-proxy-78df955549-w6fpf" Jul 08 22:55:22.266147 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.265989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4zsh\" (UniqueName: \"kubernetes.io/projected/361c10e9-fc5c-4f69-9647-ae45cebfe624-kube-api-access-z4zsh\") pod \"kube-auth-proxy-78df955549-w6fpf\" (UID: \"361c10e9-fc5c-4f69-9647-ae45cebfe624\") " pod="openshift-ingress/kube-auth-proxy-78df955549-w6fpf" Jul 08 22:55:22.266147 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.266068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/361c10e9-fc5c-4f69-9647-ae45cebfe624-tls-certs\") pod \"kube-auth-proxy-78df955549-w6fpf\" (UID: \"361c10e9-fc5c-4f69-9647-ae45cebfe624\") " pod="openshift-ingress/kube-auth-proxy-78df955549-w6fpf" Jul 08 22:55:22.312210 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.312190 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:22.367249 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.367189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/361c10e9-fc5c-4f69-9647-ae45cebfe624-tls-certs\") pod \"kube-auth-proxy-78df955549-w6fpf\" (UID: \"361c10e9-fc5c-4f69-9647-ae45cebfe624\") " pod="openshift-ingress/kube-auth-proxy-78df955549-w6fpf" Jul 08 22:55:22.367343 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.367252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/361c10e9-fc5c-4f69-9647-ae45cebfe624-tmp\") pod \"kube-auth-proxy-78df955549-w6fpf\" (UID: \"361c10e9-fc5c-4f69-9647-ae45cebfe624\") " pod="openshift-ingress/kube-auth-proxy-78df955549-w6fpf" Jul 08 22:55:22.367343 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.367284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4zsh\" (UniqueName: \"kubernetes.io/projected/361c10e9-fc5c-4f69-9647-ae45cebfe624-kube-api-access-z4zsh\") pod \"kube-auth-proxy-78df955549-w6fpf\" (UID: \"361c10e9-fc5c-4f69-9647-ae45cebfe624\") " pod="openshift-ingress/kube-auth-proxy-78df955549-w6fpf" Jul 08 22:55:22.369532 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.369514 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/361c10e9-fc5c-4f69-9647-ae45cebfe624-tmp\") pod \"kube-auth-proxy-78df955549-w6fpf\" (UID: \"361c10e9-fc5c-4f69-9647-ae45cebfe624\") " pod="openshift-ingress/kube-auth-proxy-78df955549-w6fpf" Jul 08 22:55:22.369703 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.369686 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/361c10e9-fc5c-4f69-9647-ae45cebfe624-tls-certs\") pod \"kube-auth-proxy-78df955549-w6fpf\" (UID: \"361c10e9-fc5c-4f69-9647-ae45cebfe624\") " pod="openshift-ingress/kube-auth-proxy-78df955549-w6fpf" Jul 08 22:55:22.383702 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.383681 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4zsh\" (UniqueName: \"kubernetes.io/projected/361c10e9-fc5c-4f69-9647-ae45cebfe624-kube-api-access-z4zsh\") pod \"kube-auth-proxy-78df955549-w6fpf\" (UID: \"361c10e9-fc5c-4f69-9647-ae45cebfe624\") " pod="openshift-ingress/kube-auth-proxy-78df955549-w6fpf" Jul 08 22:55:22.466001 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.465972 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:22.526054 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.526027 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78df955549-w6fpf" Jul 08 22:55:22.559674 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.559640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:22.660043 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:22.660019 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78df955549-w6fpf"] Jul 08 22:55:22.662469 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:55:22.662445 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361c10e9_fc5c_4f69_9647_ae45cebfe624.slice/crio-28cd902fffe20ffacfbc1aa7eb9143f26ac31a420bd5c1064285589286105c4f WatchSource:0}: Error finding container 28cd902fffe20ffacfbc1aa7eb9143f26ac31a420bd5c1064285589286105c4f: Status 404 returned error can't find the container with id 28cd902fffe20ffacfbc1aa7eb9143f26ac31a420bd5c1064285589286105c4f Jul 08 22:55:23.004799 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:23.004765 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:23.093921 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:23.093880 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:23.212571 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:23.212524 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:23.305895 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:23.305811 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78df955549-w6fpf" event={"ID":"361c10e9-fc5c-4f69-9647-ae45cebfe624","Type":"ContainerStarted","Data":"28cd902fffe20ffacfbc1aa7eb9143f26ac31a420bd5c1064285589286105c4f"} Jul 08 22:55:23.460026 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:23.459991 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:23.550951 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:23.550921 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:24.350879 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:24.350852 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:25.317610 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:25.317583 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78df955549-w6fpf" event={"ID":"361c10e9-fc5c-4f69-9647-ae45cebfe624","Type":"ContainerStarted","Data":"ada490ef90a73409b7da89859bb637b8ac9904990379d86829776a4f33fb8c3e"} Jul 08 22:55:25.332827 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:25.332781 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78df955549-w6fpf" podStartSLOduration=0.762556844 podStartE2EDuration="3.332766371s" totalImagesPullingTime="2.570209527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:55:25.331304923 +0000 UTC m=+519.586752627" watchObservedRunningTime="2026-07-08 22:55:25.332766371 +0000 UTC m=+519.588214074" Jul 08 22:55:28.922648 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:28.922617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:28.978782 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:28.978753 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:29.495167 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:29.495138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:29.691614 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:29.691581 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:29.809550 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:29.809516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:31.748474 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:31.748438 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:31.846187 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:31.846153 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:32.100842 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:32.100757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:32.396717 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:32.396630 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:32.994543 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:32.994515 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:33.102361 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:33.102334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:33.219264 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:33.219236 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:35.932804 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:35.932769 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:40.483125 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:40.483026 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:40.582643 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:40.582614 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:45.791267 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:45.791233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:55:45.839449 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:55:45.839424 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:56:23.225333 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:23.225301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:56:23.317723 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:23.317688 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:56:23.393502 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:23.393471 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:56:29.303965 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:29.303928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:56:31.979159 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:31.979119 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:56:32.037952 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:32.037925 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:56:33.936569 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:33.936537 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:56:46.200557 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:46.200529 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 22:56:46.204414 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:46.204388 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 22:56:46.211521 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:46.211495 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 22:56:46.215166 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:46.215146 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 22:56:52.127159 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:52.127111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:56:53.375379 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:53.375346 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:56:56.736554 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:56.736518 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:56:58.919540 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:58.919502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:56:59.024033 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:59.024002 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:56:59.792029 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:56:59.791994 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:00.744676 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:00.744644 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:00.982774 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:00.982744 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:02.954539 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:02.954510 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:03.171600 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:03.171569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:04.639436 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:04.639400 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2x6d"] Jul 08 22:57:04.643010 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:04.642986 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2x6d" Jul 08 22:57:04.646840 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:04.646823 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 22:57:04.646943 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:04.646876 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 22:57:04.646943 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:04.646899 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 08 22:57:04.647836 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:04.647816 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-lv5mc\"" Jul 08 22:57:04.656244 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:04.656222 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2x6d"] Jul 08 22:57:04.716509 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:04.716474 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljc52\" (UniqueName: \"kubernetes.io/projected/87168efd-e1a0-42e6-b466-10d867c4ed0e-kube-api-access-ljc52\") pod \"dns-operator-controller-manager-864dd78c4c-d2x6d\" (UID: \"87168efd-e1a0-42e6-b466-10d867c4ed0e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2x6d" Jul 08 22:57:04.817478 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:04.817451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljc52\" (UniqueName: \"kubernetes.io/projected/87168efd-e1a0-42e6-b466-10d867c4ed0e-kube-api-access-ljc52\") pod \"dns-operator-controller-manager-864dd78c4c-d2x6d\" (UID: \"87168efd-e1a0-42e6-b466-10d867c4ed0e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2x6d" Jul 08 22:57:04.830183 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:04.830160 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljc52\" (UniqueName: \"kubernetes.io/projected/87168efd-e1a0-42e6-b466-10d867c4ed0e-kube-api-access-ljc52\") pod \"dns-operator-controller-manager-864dd78c4c-d2x6d\" (UID: \"87168efd-e1a0-42e6-b466-10d867c4ed0e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2x6d" Jul 08 22:57:04.958736 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:04.958666 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2x6d" Jul 08 22:57:05.102839 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:05.102808 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2x6d"] Jul 08 22:57:05.105519 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:57:05.105490 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87168efd_e1a0_42e6_b466_10d867c4ed0e.slice/crio-9af75215be7783a865e81e9850cab56e148283836b9472f79ddfad67fb47633d WatchSource:0}: Error finding container 9af75215be7783a865e81e9850cab56e148283836b9472f79ddfad67fb47633d: Status 404 returned error can't find the container with id 9af75215be7783a865e81e9850cab56e148283836b9472f79ddfad67fb47633d Jul 08 22:57:05.107588 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:05.107570 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 22:57:05.783854 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:05.783821 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2x6d" event={"ID":"87168efd-e1a0-42e6-b466-10d867c4ed0e","Type":"ContainerStarted","Data":"9af75215be7783a865e81e9850cab56e148283836b9472f79ddfad67fb47633d"} Jul 08 22:57:05.957994 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:05.957959 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:06.057258 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:06.057179 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:06.340315 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:06.340236 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:06.441977 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:06.441937 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:07.795794 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:07.795757 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2x6d" event={"ID":"87168efd-e1a0-42e6-b466-10d867c4ed0e","Type":"ContainerStarted","Data":"273e20103bb63ef4711e665cfdb6010a464dc19e41cd5aee41dd4849934879b7"} Jul 08 22:57:07.796221 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:07.795869 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2x6d" Jul 08 22:57:07.815344 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:07.815297 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2x6d" podStartSLOduration=1.469289398 podStartE2EDuration="3.815282533s" totalImagesPullingTime="2.345993135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:57:07.8129694 +0000 UTC m=+622.068417103" watchObservedRunningTime="2026-07-08 22:57:07.815282533 +0000 UTC m=+622.070730236" Jul 08 22:57:09.719250 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:09.719216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:09.858445 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:09.858412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:10.057799 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:10.057766 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:10.151871 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:10.151838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:10.207353 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:10.207320 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:11.144388 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:11.144351 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:11.245901 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:11.245870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:18.355456 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:18.355427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:18.462728 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:18.462697 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:18.802445 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:18.802418 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2x6d" Jul 08 22:57:19.965727 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:19.965694 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t"] Jul 08 22:57:19.969003 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:19.968988 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t" Jul 08 22:57:19.971556 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:19.971534 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9wrlv\"" Jul 08 22:57:19.982700 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:19.982681 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t"] Jul 08 22:57:20.048023 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.047997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c57d2791-2efe-42e3-b8a8-f0e87c74343c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7h67t\" (UID: \"c57d2791-2efe-42e3-b8a8-f0e87c74343c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t" Jul 08 22:57:20.048202 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.048075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pncct\" (UniqueName: \"kubernetes.io/projected/c57d2791-2efe-42e3-b8a8-f0e87c74343c-kube-api-access-pncct\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7h67t\" (UID: \"c57d2791-2efe-42e3-b8a8-f0e87c74343c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t" Jul 08 22:57:20.148523 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.148490 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c57d2791-2efe-42e3-b8a8-f0e87c74343c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7h67t\" (UID: \"c57d2791-2efe-42e3-b8a8-f0e87c74343c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t" Jul 08 22:57:20.148731 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.148555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pncct\" (UniqueName: \"kubernetes.io/projected/c57d2791-2efe-42e3-b8a8-f0e87c74343c-kube-api-access-pncct\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7h67t\" (UID: \"c57d2791-2efe-42e3-b8a8-f0e87c74343c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t" Jul 08 22:57:20.148879 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.148857 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c57d2791-2efe-42e3-b8a8-f0e87c74343c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7h67t\" (UID: \"c57d2791-2efe-42e3-b8a8-f0e87c74343c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t" Jul 08 22:57:20.159282 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.159259 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pncct\" (UniqueName: \"kubernetes.io/projected/c57d2791-2efe-42e3-b8a8-f0e87c74343c-kube-api-access-pncct\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7h67t\" (UID: \"c57d2791-2efe-42e3-b8a8-f0e87c74343c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t" Jul 08 22:57:20.279273 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.279248 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t" Jul 08 22:57:20.414703 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.414675 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t"] Jul 08 22:57:20.416792 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:57:20.416759 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57d2791_2efe_42e3_b8a8_f0e87c74343c.slice/crio-91ab6efdeadb5914df27e99ed52c246a7248fd521b37dc45fbb067760844687f WatchSource:0}: Error finding container 91ab6efdeadb5914df27e99ed52c246a7248fd521b37dc45fbb067760844687f: Status 404 returned error can't find the container with id 91ab6efdeadb5914df27e99ed52c246a7248fd521b37dc45fbb067760844687f Jul 08 22:57:20.580696 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.580623 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t"] Jul 08 22:57:20.587110 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.587063 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t"] Jul 08 22:57:20.607659 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.607630 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv"] Jul 08 22:57:20.612132 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.612107 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" Jul 08 22:57:20.625758 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.625731 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv"] Jul 08 22:57:20.756993 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.756925 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw8ng\" (UniqueName: \"kubernetes.io/projected/f121aa3c-feb1-4c32-b3e7-74172429034d-kube-api-access-lw8ng\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wrgsv\" (UID: \"f121aa3c-feb1-4c32-b3e7-74172429034d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" Jul 08 22:57:20.757233 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.757072 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f121aa3c-feb1-4c32-b3e7-74172429034d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wrgsv\" (UID: \"f121aa3c-feb1-4c32-b3e7-74172429034d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" Jul 08 22:57:20.803898 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.803870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:20.858448 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.858387 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw8ng\" (UniqueName: \"kubernetes.io/projected/f121aa3c-feb1-4c32-b3e7-74172429034d-kube-api-access-lw8ng\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wrgsv\" (UID: \"f121aa3c-feb1-4c32-b3e7-74172429034d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" Jul 08 22:57:20.858570 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.858458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f121aa3c-feb1-4c32-b3e7-74172429034d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wrgsv\" (UID: \"f121aa3c-feb1-4c32-b3e7-74172429034d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" Jul 08 22:57:20.858806 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.858788 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f121aa3c-feb1-4c32-b3e7-74172429034d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wrgsv\" (UID: \"f121aa3c-feb1-4c32-b3e7-74172429034d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" Jul 08 22:57:20.866623 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.866600 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw8ng\" (UniqueName: \"kubernetes.io/projected/f121aa3c-feb1-4c32-b3e7-74172429034d-kube-api-access-lw8ng\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wrgsv\" (UID: \"f121aa3c-feb1-4c32-b3e7-74172429034d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" Jul 08 22:57:20.904206 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.904181 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:20.925261 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:20.925238 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" Jul 08 22:57:21.068165 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:21.068142 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv"] Jul 08 22:57:21.074961 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:57:21.074934 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf121aa3c_feb1_4c32_b3e7_74172429034d.slice/crio-869268520a2cbaaf8707655fe5b6f4e9bc9728d177d35c2f8d4dd7294ac38dc2 WatchSource:0}: Error finding container 869268520a2cbaaf8707655fe5b6f4e9bc9728d177d35c2f8d4dd7294ac38dc2: Status 404 returned error can't find the container with id 869268520a2cbaaf8707655fe5b6f4e9bc9728d177d35c2f8d4dd7294ac38dc2 Jul 08 22:57:21.863163 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:21.863116 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" event={"ID":"f121aa3c-feb1-4c32-b3e7-74172429034d","Type":"ContainerStarted","Data":"869268520a2cbaaf8707655fe5b6f4e9bc9728d177d35c2f8d4dd7294ac38dc2"} Jul 08 22:57:25.889801 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:25.889736 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t" podUID="c57d2791-2efe-42e3-b8a8-f0e87c74343c" containerName="manager" containerID="cri-o://ce63709c412d89742b6733e2751f83651a13052e00ec9c67fcbcb8dc3de9e957" gracePeriod=2 Jul 08 22:57:25.891376 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:25.891349 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" event={"ID":"f121aa3c-feb1-4c32-b3e7-74172429034d","Type":"ContainerStarted","Data":"7475282da8a3287194e2c5074c37c8e58817b11528e463d0155430ae6d698c9a"} Jul 08 22:57:25.891501 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:25.891432 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" Jul 08 22:57:25.914851 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:25.914800 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" podStartSLOduration=1.7968919749999999 podStartE2EDuration="5.914785676s" totalImagesPullingTime="4.117893701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:57:25.912764975 +0000 UTC m=+640.168212681" watchObservedRunningTime="2026-07-08 22:57:25.914785676 +0000 UTC m=+640.170233388" Jul 08 22:57:26.135622 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.135597 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t" Jul 08 22:57:26.138022 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:57:26.137995 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7h67t\" is forbidden: User \"system:node:ip-10-0-143-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-210.ec2.internal' and this object" podUID="c57d2791-2efe-42e3-b8a8-f0e87c74343c" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t" Jul 08 22:57:26.205277 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.205204 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c57d2791-2efe-42e3-b8a8-f0e87c74343c-extensions-socket-volume\") pod \"c57d2791-2efe-42e3-b8a8-f0e87c74343c\" (UID: \"c57d2791-2efe-42e3-b8a8-f0e87c74343c\") " Jul 08 22:57:26.205399 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.205325 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pncct\" (UniqueName: \"kubernetes.io/projected/c57d2791-2efe-42e3-b8a8-f0e87c74343c-kube-api-access-pncct\") pod \"c57d2791-2efe-42e3-b8a8-f0e87c74343c\" (UID: \"c57d2791-2efe-42e3-b8a8-f0e87c74343c\") " Jul 08 22:57:26.205484 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.205461 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57d2791-2efe-42e3-b8a8-f0e87c74343c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c57d2791-2efe-42e3-b8a8-f0e87c74343c" (UID: "c57d2791-2efe-42e3-b8a8-f0e87c74343c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 22:57:26.207388 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.207367 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57d2791-2efe-42e3-b8a8-f0e87c74343c-kube-api-access-pncct" (OuterVolumeSpecName: "kube-api-access-pncct") pod "c57d2791-2efe-42e3-b8a8-f0e87c74343c" (UID: "c57d2791-2efe-42e3-b8a8-f0e87c74343c"). InnerVolumeSpecName "kube-api-access-pncct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 22:57:26.269803 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:57:26.269761 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7h67t\" is forbidden: User \"system:node:ip-10-0-143-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-210.ec2.internal' and this object" podUID="c57d2791-2efe-42e3-b8a8-f0e87c74343c" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t" Jul 08 22:57:26.272757 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.272734 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c57d2791-2efe-42e3-b8a8-f0e87c74343c" path="/var/lib/kubelet/pods/c57d2791-2efe-42e3-b8a8-f0e87c74343c/volumes" Jul 08 22:57:26.305828 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.305805 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c57d2791-2efe-42e3-b8a8-f0e87c74343c-extensions-socket-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:57:26.305828 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.305827 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pncct\" (UniqueName: \"kubernetes.io/projected/c57d2791-2efe-42e3-b8a8-f0e87c74343c-kube-api-access-pncct\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:57:26.556693 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.556663 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:26.621456 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.621428 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:26.896951 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.896870 2564 generic.go:356] "Generic (PLEG): container finished" podID="c57d2791-2efe-42e3-b8a8-f0e87c74343c" containerID="ce63709c412d89742b6733e2751f83651a13052e00ec9c67fcbcb8dc3de9e957" exitCode=2 Jul 08 22:57:26.896951 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.896921 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t" Jul 08 22:57:26.897401 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.896962 2564 scope.go:117] "RemoveContainer" containerID="ce63709c412d89742b6733e2751f83651a13052e00ec9c67fcbcb8dc3de9e957" Jul 08 22:57:26.901629 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:57:26.901589 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7h67t\" is forbidden: User \"system:node:ip-10-0-143-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-210.ec2.internal' and this object" podUID="c57d2791-2efe-42e3-b8a8-f0e87c74343c" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7h67t" Jul 08 22:57:26.905528 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.905514 2564 scope.go:117] "RemoveContainer" containerID="ce63709c412d89742b6733e2751f83651a13052e00ec9c67fcbcb8dc3de9e957" Jul 08 22:57:26.905786 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:57:26.905769 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce63709c412d89742b6733e2751f83651a13052e00ec9c67fcbcb8dc3de9e957\": container with ID starting with ce63709c412d89742b6733e2751f83651a13052e00ec9c67fcbcb8dc3de9e957 not found: ID does not exist" containerID="ce63709c412d89742b6733e2751f83651a13052e00ec9c67fcbcb8dc3de9e957" Jul 08 22:57:26.905828 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:26.905796 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce63709c412d89742b6733e2751f83651a13052e00ec9c67fcbcb8dc3de9e957"} err="failed to get container status \"ce63709c412d89742b6733e2751f83651a13052e00ec9c67fcbcb8dc3de9e957\": rpc error: code = NotFound desc = could not find container \"ce63709c412d89742b6733e2751f83651a13052e00ec9c67fcbcb8dc3de9e957\": container with ID starting with ce63709c412d89742b6733e2751f83651a13052e00ec9c67fcbcb8dc3de9e957 not found: ID does not exist" Jul 08 22:57:27.161560 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:27.161454 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:27.261796 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:27.261766 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:30.757335 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:30.757303 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:30.815283 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:30.815256 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:36.899298 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:36.899267 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" Jul 08 22:57:38.563192 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:38.563156 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:38.663516 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:38.663466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:40.558121 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:40.558070 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:40.658812 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:40.658781 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:49.134526 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:49.134492 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv"] Jul 08 22:57:49.134991 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:49.134706 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" podUID="f121aa3c-feb1-4c32-b3e7-74172429034d" containerName="manager" containerID="cri-o://7475282da8a3287194e2c5074c37c8e58817b11528e463d0155430ae6d698c9a" gracePeriod=10 Jul 08 22:57:49.383291 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:49.383266 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" Jul 08 22:57:49.407347 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:49.407282 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f121aa3c-feb1-4c32-b3e7-74172429034d-extensions-socket-volume\") pod \"f121aa3c-feb1-4c32-b3e7-74172429034d\" (UID: \"f121aa3c-feb1-4c32-b3e7-74172429034d\") " Jul 08 22:57:49.407486 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:49.407374 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw8ng\" (UniqueName: \"kubernetes.io/projected/f121aa3c-feb1-4c32-b3e7-74172429034d-kube-api-access-lw8ng\") pod \"f121aa3c-feb1-4c32-b3e7-74172429034d\" (UID: \"f121aa3c-feb1-4c32-b3e7-74172429034d\") " Jul 08 22:57:49.407663 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:49.407636 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f121aa3c-feb1-4c32-b3e7-74172429034d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f121aa3c-feb1-4c32-b3e7-74172429034d" (UID: "f121aa3c-feb1-4c32-b3e7-74172429034d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 22:57:49.409379 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:49.409351 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f121aa3c-feb1-4c32-b3e7-74172429034d-kube-api-access-lw8ng" (OuterVolumeSpecName: "kube-api-access-lw8ng") pod "f121aa3c-feb1-4c32-b3e7-74172429034d" (UID: "f121aa3c-feb1-4c32-b3e7-74172429034d"). InnerVolumeSpecName "kube-api-access-lw8ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 22:57:49.508503 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:49.508468 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f121aa3c-feb1-4c32-b3e7-74172429034d-extensions-socket-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:57:49.508503 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:49.508496 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw8ng\" (UniqueName: \"kubernetes.io/projected/f121aa3c-feb1-4c32-b3e7-74172429034d-kube-api-access-lw8ng\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:57:50.008137 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:50.008104 2564 generic.go:356] "Generic (PLEG): container finished" podID="f121aa3c-feb1-4c32-b3e7-74172429034d" containerID="7475282da8a3287194e2c5074c37c8e58817b11528e463d0155430ae6d698c9a" exitCode=0 Jul 08 22:57:50.008137 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:50.008140 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" Jul 08 22:57:50.008360 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:50.008177 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" event={"ID":"f121aa3c-feb1-4c32-b3e7-74172429034d","Type":"ContainerDied","Data":"7475282da8a3287194e2c5074c37c8e58817b11528e463d0155430ae6d698c9a"} Jul 08 22:57:50.008360 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:50.008236 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv" event={"ID":"f121aa3c-feb1-4c32-b3e7-74172429034d","Type":"ContainerDied","Data":"869268520a2cbaaf8707655fe5b6f4e9bc9728d177d35c2f8d4dd7294ac38dc2"} Jul 08 22:57:50.008360 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:50.008263 2564 scope.go:117] "RemoveContainer" containerID="7475282da8a3287194e2c5074c37c8e58817b11528e463d0155430ae6d698c9a" Jul 08 22:57:50.022328 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:50.022307 2564 scope.go:117] "RemoveContainer" containerID="7475282da8a3287194e2c5074c37c8e58817b11528e463d0155430ae6d698c9a" Jul 08 22:57:50.022627 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:57:50.022607 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7475282da8a3287194e2c5074c37c8e58817b11528e463d0155430ae6d698c9a\": container with ID starting with 7475282da8a3287194e2c5074c37c8e58817b11528e463d0155430ae6d698c9a not found: ID does not exist" containerID="7475282da8a3287194e2c5074c37c8e58817b11528e463d0155430ae6d698c9a" Jul 08 22:57:50.022704 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:50.022640 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7475282da8a3287194e2c5074c37c8e58817b11528e463d0155430ae6d698c9a"} err="failed to get container status \"7475282da8a3287194e2c5074c37c8e58817b11528e463d0155430ae6d698c9a\": rpc error: code = NotFound desc = could not find container \"7475282da8a3287194e2c5074c37c8e58817b11528e463d0155430ae6d698c9a\": container with ID starting with 7475282da8a3287194e2c5074c37c8e58817b11528e463d0155430ae6d698c9a not found: ID does not exist" Jul 08 22:57:50.038913 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:50.038890 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv"] Jul 08 22:57:50.043785 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:50.043762 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wrgsv"] Jul 08 22:57:50.268049 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:50.267970 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f121aa3c-feb1-4c32-b3e7-74172429034d" path="/var/lib/kubelet/pods/f121aa3c-feb1-4c32-b3e7-74172429034d/volumes" Jul 08 22:57:53.257511 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:53.257480 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:57:53.562781 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:57:53.562694 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:05.487234 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:05.487200 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:05.588599 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:05.588561 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:07.064232 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.064202 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:07.177469 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.177442 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:07.243683 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.243647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:07.463288 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.463211 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 22:58:07.463579 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.463567 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57d2791-2efe-42e3-b8a8-f0e87c74343c" containerName="manager" Jul 08 22:58:07.463623 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.463580 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57d2791-2efe-42e3-b8a8-f0e87c74343c" containerName="manager" Jul 08 22:58:07.463623 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.463588 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f121aa3c-feb1-4c32-b3e7-74172429034d" containerName="manager" Jul 08 22:58:07.463623 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.463594 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f121aa3c-feb1-4c32-b3e7-74172429034d" containerName="manager" Jul 08 22:58:07.463719 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.463655 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57d2791-2efe-42e3-b8a8-f0e87c74343c" containerName="manager" Jul 08 22:58:07.463719 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.463666 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f121aa3c-feb1-4c32-b3e7-74172429034d" containerName="manager" Jul 08 22:58:07.468006 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.467985 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jsfqw" Jul 08 22:58:07.470287 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.470266 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 22:58:07.470390 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.470353 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qjss4\"" Jul 08 22:58:07.473563 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.473541 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 22:58:07.500631 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.500602 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 22:58:07.565755 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.565726 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f518cf6-b12a-4943-b210-2bcf0cc459ed-config-file\") pod \"limitador-limitador-54d6d988d5-jsfqw\" (UID: \"9f518cf6-b12a-4943-b210-2bcf0cc459ed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jsfqw" Jul 08 22:58:07.565921 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.565775 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g5n2\" (UniqueName: \"kubernetes.io/projected/9f518cf6-b12a-4943-b210-2bcf0cc459ed-kube-api-access-5g5n2\") pod \"limitador-limitador-54d6d988d5-jsfqw\" (UID: \"9f518cf6-b12a-4943-b210-2bcf0cc459ed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jsfqw" Jul 08 22:58:07.667184 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.667150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f518cf6-b12a-4943-b210-2bcf0cc459ed-config-file\") pod \"limitador-limitador-54d6d988d5-jsfqw\" (UID: \"9f518cf6-b12a-4943-b210-2bcf0cc459ed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jsfqw" Jul 08 22:58:07.667355 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.667203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g5n2\" (UniqueName: \"kubernetes.io/projected/9f518cf6-b12a-4943-b210-2bcf0cc459ed-kube-api-access-5g5n2\") pod \"limitador-limitador-54d6d988d5-jsfqw\" (UID: \"9f518cf6-b12a-4943-b210-2bcf0cc459ed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jsfqw" Jul 08 22:58:07.667775 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.667756 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f518cf6-b12a-4943-b210-2bcf0cc459ed-config-file\") pod \"limitador-limitador-54d6d988d5-jsfqw\" (UID: \"9f518cf6-b12a-4943-b210-2bcf0cc459ed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jsfqw" Jul 08 22:58:07.675449 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.675425 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g5n2\" (UniqueName: \"kubernetes.io/projected/9f518cf6-b12a-4943-b210-2bcf0cc459ed-kube-api-access-5g5n2\") pod \"limitador-limitador-54d6d988d5-jsfqw\" (UID: \"9f518cf6-b12a-4943-b210-2bcf0cc459ed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jsfqw" Jul 08 22:58:07.779120 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.779071 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jsfqw" Jul 08 22:58:07.908964 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.908938 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 22:58:07.910515 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.910457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:07.911456 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:58:07.911434 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f518cf6_b12a_4943_b210_2bcf0cc459ed.slice/crio-e2251579822d16fdd014322aa1e5b62f9c5d1eb6f4d24269ed877534f62a1f69 WatchSource:0}: Error finding container e2251579822d16fdd014322aa1e5b62f9c5d1eb6f4d24269ed877534f62a1f69: Status 404 returned error can't find the container with id e2251579822d16fdd014322aa1e5b62f9c5d1eb6f4d24269ed877534f62a1f69 Jul 08 22:58:07.970450 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:07.970423 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:08.103884 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:08.103798 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jsfqw" event={"ID":"9f518cf6-b12a-4943-b210-2bcf0cc459ed","Type":"ContainerStarted","Data":"e2251579822d16fdd014322aa1e5b62f9c5d1eb6f4d24269ed877534f62a1f69"} Jul 08 22:58:11.120466 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:11.120419 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jsfqw" event={"ID":"9f518cf6-b12a-4943-b210-2bcf0cc459ed","Type":"ContainerStarted","Data":"8aaeab4ed3ee2198a35f3bd64ac478ad71bbf29b90ef40b495ee73cad3f6efb5"} Jul 08 22:58:11.120908 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:11.120518 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jsfqw" Jul 08 22:58:11.136538 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:11.136470 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-jsfqw" podStartSLOduration=1.621385776 podStartE2EDuration="4.136455501s" totalImagesPullingTime="2.515069725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:58:11.135458016 +0000 UTC m=+685.390905720" watchObservedRunningTime="2026-07-08 22:58:11.136455501 +0000 UTC m=+685.391903206" Jul 08 22:58:22.126584 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:22.126556 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jsfqw" Jul 08 22:58:26.553779 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:26.553746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:26.651539 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:26.651504 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:27.002073 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:27.002041 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:38.740372 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:38.740288 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:39.127775 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:39.127737 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:39.216115 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:39.216071 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59b9985cbc-nsdbm"] Jul 08 22:58:39.221241 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:39.221224 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59b9985cbc-nsdbm" Jul 08 22:58:39.223696 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:39.223673 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dmwqc\"" Jul 08 22:58:39.226863 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:39.226839 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59b9985cbc-nsdbm"] Jul 08 22:58:39.235369 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:39.235347 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97ccl\" (UniqueName: \"kubernetes.io/projected/280b082b-b322-41de-a144-6a0fff94e268-kube-api-access-97ccl\") pod \"authorino-59b9985cbc-nsdbm\" (UID: \"280b082b-b322-41de-a144-6a0fff94e268\") " pod="kuadrant-system/authorino-59b9985cbc-nsdbm" Jul 08 22:58:39.318725 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:39.318700 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59b9985cbc-nsdbm"] Jul 08 22:58:39.318951 ip-10-0-143-210 kubenswrapper[2564]: E0708 22:58:39.318934 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-97ccl], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-59b9985cbc-nsdbm" podUID="280b082b-b322-41de-a144-6a0fff94e268" Jul 08 22:58:39.336473 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:39.336450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97ccl\" (UniqueName: \"kubernetes.io/projected/280b082b-b322-41de-a144-6a0fff94e268-kube-api-access-97ccl\") pod \"authorino-59b9985cbc-nsdbm\" (UID: \"280b082b-b322-41de-a144-6a0fff94e268\") " pod="kuadrant-system/authorino-59b9985cbc-nsdbm" Jul 08 22:58:39.349734 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:39.349705 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97ccl\" (UniqueName: \"kubernetes.io/projected/280b082b-b322-41de-a144-6a0fff94e268-kube-api-access-97ccl\") pod \"authorino-59b9985cbc-nsdbm\" (UID: \"280b082b-b322-41de-a144-6a0fff94e268\") " pod="kuadrant-system/authorino-59b9985cbc-nsdbm" Jul 08 22:58:40.266635 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:40.266599 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59b9985cbc-nsdbm" Jul 08 22:58:40.274448 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:40.274424 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59b9985cbc-nsdbm" Jul 08 22:58:40.344372 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:40.344342 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97ccl\" (UniqueName: \"kubernetes.io/projected/280b082b-b322-41de-a144-6a0fff94e268-kube-api-access-97ccl\") pod \"280b082b-b322-41de-a144-6a0fff94e268\" (UID: \"280b082b-b322-41de-a144-6a0fff94e268\") " Jul 08 22:58:40.346398 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:40.346371 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280b082b-b322-41de-a144-6a0fff94e268-kube-api-access-97ccl" (OuterVolumeSpecName: "kube-api-access-97ccl") pod "280b082b-b322-41de-a144-6a0fff94e268" (UID: "280b082b-b322-41de-a144-6a0fff94e268"). InnerVolumeSpecName "kube-api-access-97ccl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 22:58:40.445234 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:40.445200 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97ccl\" (UniqueName: \"kubernetes.io/projected/280b082b-b322-41de-a144-6a0fff94e268-kube-api-access-97ccl\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 22:58:41.270898 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:41.270868 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59b9985cbc-nsdbm" Jul 08 22:58:41.305168 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:41.305130 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59b9985cbc-nsdbm"] Jul 08 22:58:41.305638 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:41.305613 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59b9985cbc-nsdbm"] Jul 08 22:58:42.267654 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:42.267623 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="280b082b-b322-41de-a144-6a0fff94e268" path="/var/lib/kubelet/pods/280b082b-b322-41de-a144-6a0fff94e268/volumes" Jul 08 22:58:44.963301 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:44.963241 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:45.055573 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:45.055537 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:45.128037 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:45.128006 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:45.435560 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:45.435530 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:49.223679 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:49.223642 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:49.327274 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:49.327244 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:58.552068 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:58.552031 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:58:58.611429 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:58:58.611401 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:59:22.426132 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:22.426075 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:59:22.906864 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:22.906826 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:59:23.084955 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:23.084924 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:59:23.184364 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:23.184287 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:59:23.329479 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:23.329446 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:59:23.435091 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:23.435001 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:59:23.491151 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:23.491116 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 22:59:34.689310 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:34.689274 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:59:34.791033 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:34.790991 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:59:47.083564 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.083526 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w"] Jul 08 22:59:47.087004 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.086982 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.089450 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.089429 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 22:59:47.090641 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.090614 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 08 22:59:47.090737 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.090639 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qxrkl\"" Jul 08 22:59:47.090737 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.090662 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 22:59:47.094534 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.094513 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w"] Jul 08 22:59:47.176781 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.176748 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:59:47.224676 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.224648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bc58678-984d-4964-8300-5a26514f24a7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.224815 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.224680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bc58678-984d-4964-8300-5a26514f24a7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.224815 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.224705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc58678-984d-4964-8300-5a26514f24a7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.224815 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.224760 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bc58678-984d-4964-8300-5a26514f24a7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.224815 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.224790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhktg\" (UniqueName: \"kubernetes.io/projected/2bc58678-984d-4964-8300-5a26514f24a7-kube-api-access-dhktg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.225016 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.224864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bc58678-984d-4964-8300-5a26514f24a7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.325538 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.325509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bc58678-984d-4964-8300-5a26514f24a7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.325716 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.325542 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bc58678-984d-4964-8300-5a26514f24a7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.325716 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.325566 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc58678-984d-4964-8300-5a26514f24a7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.325716 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.325589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bc58678-984d-4964-8300-5a26514f24a7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.325886 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.325733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhktg\" (UniqueName: \"kubernetes.io/projected/2bc58678-984d-4964-8300-5a26514f24a7-kube-api-access-dhktg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.325886 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.325787 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bc58678-984d-4964-8300-5a26514f24a7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.326036 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.326014 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bc58678-984d-4964-8300-5a26514f24a7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.326292 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.326042 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc58678-984d-4964-8300-5a26514f24a7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.326292 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.326138 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bc58678-984d-4964-8300-5a26514f24a7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.328373 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.328347 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bc58678-984d-4964-8300-5a26514f24a7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.328720 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.328698 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bc58678-984d-4964-8300-5a26514f24a7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.333147 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.333128 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhktg\" (UniqueName: \"kubernetes.io/projected/2bc58678-984d-4964-8300-5a26514f24a7-kube-api-access-dhktg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w\" (UID: \"2bc58678-984d-4964-8300-5a26514f24a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.398560 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.398480 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:47.530074 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.530044 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w"] Jul 08 22:59:47.531739 ip-10-0-143-210 kubenswrapper[2564]: W0708 22:59:47.531709 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc58678_984d_4964_8300_5a26514f24a7.slice/crio-d44a14b5c14067fbe3fb6eba040d947a06446c936073bf20474bc0f45ecb480f WatchSource:0}: Error finding container d44a14b5c14067fbe3fb6eba040d947a06446c936073bf20474bc0f45ecb480f: Status 404 returned error can't find the container with id d44a14b5c14067fbe3fb6eba040d947a06446c936073bf20474bc0f45ecb480f Jul 08 22:59:47.572652 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.572621 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" event={"ID":"2bc58678-984d-4964-8300-5a26514f24a7","Type":"ContainerStarted","Data":"d44a14b5c14067fbe3fb6eba040d947a06446c936073bf20474bc0f45ecb480f"} Jul 08 22:59:47.678071 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:47.677997 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 22:59:48.578431 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:48.578407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" event={"ID":"2bc58678-984d-4964-8300-5a26514f24a7","Type":"ContainerStarted","Data":"67e8b94efd92305ad6148bbf10a467c0d01cfe0c61e09752897ea2d4c66fe2d7"} Jul 08 22:59:48.598304 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:48.598253 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" podStartSLOduration=0.620544978 podStartE2EDuration="1.598235888s" totalImagesPullingTime="977.69091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 22:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 22:59:48.593933581 +0000 UTC m=+782.849381284" watchObservedRunningTime="2026-07-08 22:59:48.598235888 +0000 UTC m=+782.853683591" Jul 08 22:59:54.109727 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:54.109695 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 22:59:54.988844 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:54.988799 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 22:59:57.398638 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:57.398606 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:57.399133 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:57.398686 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:57.411128 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:57.411107 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 22:59:57.635953 ip-10-0-143-210 kubenswrapper[2564]: I0708 22:59:57.635930 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w" Jul 08 23:00:09.333141 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:09.333049 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:00:10.472759 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:10.472728 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:00:40.481387 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.481354 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv"] Jul 08 23:00:40.486661 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.486639 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.489236 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.489216 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 08 23:00:40.495937 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.495916 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv"] Jul 08 23:00:40.515285 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.515257 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:00:40.594309 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.594268 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.594309 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.594313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.594517 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.594417 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.594517 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.594456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nw5r\" (UniqueName: \"kubernetes.io/projected/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-kube-api-access-5nw5r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.594517 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.594483 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.594652 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.594548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.695340 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.695303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.695527 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.695344 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.695527 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.695401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.695527 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.695432 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nw5r\" (UniqueName: \"kubernetes.io/projected/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-kube-api-access-5nw5r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.695527 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.695457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.695527 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.695492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.695795 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.695713 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.695854 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.695832 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.695915 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.695886 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.697556 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.697531 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.697806 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.697789 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.703147 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.703124 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nw5r\" (UniqueName: \"kubernetes.io/projected/ff6984e4-3311-4b8f-8e5a-4421eefeba9e-kube-api-access-5nw5r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv\" (UID: \"ff6984e4-3311-4b8f-8e5a-4421eefeba9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.796931 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.796902 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:40.935369 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:40.935338 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv"] Jul 08 23:00:40.937566 ip-10-0-143-210 kubenswrapper[2564]: W0708 23:00:40.937530 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6984e4_3311_4b8f_8e5a_4421eefeba9e.slice/crio-d9339afe05677303617dac3b71084f5543ab57858ed23257896fc661f7992b2d WatchSource:0}: Error finding container d9339afe05677303617dac3b71084f5543ab57858ed23257896fc661f7992b2d: Status 404 returned error can't find the container with id d9339afe05677303617dac3b71084f5543ab57858ed23257896fc661f7992b2d Jul 08 23:00:41.832568 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:41.832533 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" event={"ID":"ff6984e4-3311-4b8f-8e5a-4421eefeba9e","Type":"ContainerStarted","Data":"70607388d5ed943a7d66b2d074d5cbc8297de7a39fd07195a82b186afff647eb"} Jul 08 23:00:41.832943 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:41.832586 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" event={"ID":"ff6984e4-3311-4b8f-8e5a-4421eefeba9e","Type":"ContainerStarted","Data":"d9339afe05677303617dac3b71084f5543ab57858ed23257896fc661f7992b2d"} Jul 08 23:00:41.850133 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:41.850077 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" podStartSLOduration=1.65464626 podStartE2EDuration="1.85006189s" totalImagesPullingTime="195.41563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:00:41.848211765 +0000 UTC m=+836.103659469" watchObservedRunningTime="2026-07-08 23:00:41.85006189 +0000 UTC m=+836.105509592" Jul 08 23:00:42.378617 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:42.378580 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:00:50.797320 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:50.797268 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:50.797817 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:50.797346 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:50.809867 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:50.809843 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:50.887871 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:50.887843 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv" Jul 08 23:00:59.276590 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:00:59.276554 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:01:01.182677 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:01.182644 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:01:04.978246 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:04.978214 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh"] Jul 08 23:01:04.982228 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:04.982208 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:04.984668 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:04.984651 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 08 23:01:04.991584 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:04.991562 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh"] Jul 08 23:01:05.012678 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.012653 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:01:05.109960 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.109931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7799ee64-f6f8-44ce-8147-eaa844c87254-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.110113 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.109965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7799ee64-f6f8-44ce-8147-eaa844c87254-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.110113 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.110050 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7799ee64-f6f8-44ce-8147-eaa844c87254-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.110113 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.110070 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq4sf\" (UniqueName: \"kubernetes.io/projected/7799ee64-f6f8-44ce-8147-eaa844c87254-kube-api-access-mq4sf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.110247 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.110136 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7799ee64-f6f8-44ce-8147-eaa844c87254-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.110247 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.110156 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7799ee64-f6f8-44ce-8147-eaa844c87254-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.211074 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.211045 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7799ee64-f6f8-44ce-8147-eaa844c87254-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.211209 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.211099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq4sf\" (UniqueName: \"kubernetes.io/projected/7799ee64-f6f8-44ce-8147-eaa844c87254-kube-api-access-mq4sf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.211209 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.211160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7799ee64-f6f8-44ce-8147-eaa844c87254-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.211209 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.211189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7799ee64-f6f8-44ce-8147-eaa844c87254-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.211393 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.211217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7799ee64-f6f8-44ce-8147-eaa844c87254-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.211393 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.211248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7799ee64-f6f8-44ce-8147-eaa844c87254-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.211585 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.211565 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7799ee64-f6f8-44ce-8147-eaa844c87254-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.211652 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.211586 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7799ee64-f6f8-44ce-8147-eaa844c87254-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.211652 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.211610 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7799ee64-f6f8-44ce-8147-eaa844c87254-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.213560 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.213536 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7799ee64-f6f8-44ce-8147-eaa844c87254-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.213787 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.213770 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7799ee64-f6f8-44ce-8147-eaa844c87254-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.218584 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.218560 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq4sf\" (UniqueName: \"kubernetes.io/projected/7799ee64-f6f8-44ce-8147-eaa844c87254-kube-api-access-mq4sf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh\" (UID: \"7799ee64-f6f8-44ce-8147-eaa844c87254\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.291706 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.291677 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:05.424563 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.424539 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh"] Jul 08 23:01:05.426686 ip-10-0-143-210 kubenswrapper[2564]: W0708 23:01:05.426657 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7799ee64_f6f8_44ce_8147_eaa844c87254.slice/crio-61b61e964eb26922b5171c72abee50751b49fc70212447e20f5f6471cb69d687 WatchSource:0}: Error finding container 61b61e964eb26922b5171c72abee50751b49fc70212447e20f5f6471cb69d687: Status 404 returned error can't find the container with id 61b61e964eb26922b5171c72abee50751b49fc70212447e20f5f6471cb69d687 Jul 08 23:01:05.950325 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.950242 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" event={"ID":"7799ee64-f6f8-44ce-8147-eaa844c87254","Type":"ContainerStarted","Data":"92588ba27ca4e3d47d068f7118bde22594bde24b1b71889a3579d7c64043475e"} Jul 08 23:01:05.950325 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.950288 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" event={"ID":"7799ee64-f6f8-44ce-8147-eaa844c87254","Type":"ContainerStarted","Data":"61b61e964eb26922b5171c72abee50751b49fc70212447e20f5f6471cb69d687"} Jul 08 23:01:05.967663 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:05.967611 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" podStartSLOduration=1.754105595 podStartE2EDuration="1.967598018s" totalImagesPullingTime="213.492423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:01:05.96623106 +0000 UTC m=+860.221678754" watchObservedRunningTime="2026-07-08 23:01:05.967598018 +0000 UTC m=+860.223045720" Jul 08 23:01:12.028446 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:12.028413 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:01:12.077270 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:12.077244 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:01:14.279574 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:14.279538 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:01:15.292435 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:15.292408 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:15.292801 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:15.292448 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:15.304688 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:15.304658 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:16.009750 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:16.009721 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh" Jul 08 23:01:21.373938 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:21.373905 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:01:46.238832 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:46.238793 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:01:46.242224 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:46.242200 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 23:01:46.243963 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:46.243941 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:01:46.247562 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:01:46.247547 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 23:02:05.497456 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:02:05.497424 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:02:10.967375 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:02:10.967339 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:02:11.041553 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:02:11.041499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:02:12.050944 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:02:12.050909 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:02:12.151710 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:02:12.151677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:02:37.308924 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:02:37.308888 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:02:39.975173 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:02:39.975136 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:02:49.477439 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:02:49.477405 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:03:01.382157 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:03:01.382124 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:03:05.929991 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:03:05.929955 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:03:05.986262 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:03:05.986234 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:03:09.806372 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:03:09.806288 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:03:19.873186 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:03:19.873152 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:03:21.777311 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:03:21.777279 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:03:23.618616 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:03:23.618587 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:03:46.679676 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:03:46.679645 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:03:58.675947 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:03:58.675910 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:04:03.202312 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:04:03.202278 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:04:03.301214 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:04:03.301176 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:04:03.410917 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:04:03.410882 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:04:09.029748 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:04:09.029714 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:04:09.099291 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:04:09.099262 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:04:09.176708 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:04:09.176678 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:04:09.386610 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:04:09.386498 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:04:09.920107 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:04:09.920046 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:04:15.063687 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:04:15.063653 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:04:15.123220 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:04:15.123179 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:04:31.278493 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:04:31.278457 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:04:41.281106 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:04:41.281004 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:04:50.086540 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:04:50.086503 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:05:01.375308 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:05:01.375272 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:05:01.538487 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:05:01.538433 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:05:11.089321 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:05:11.089285 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:05:23.187141 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:05:23.187104 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:05:34.080145 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:05:34.080104 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:05:44.072608 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:05:44.072575 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:06:05.683165 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:06:05.683135 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:06:14.771930 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:06:14.771846 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:06:38.381567 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:06:38.381537 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:06:46.275322 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:06:46.275296 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:06:46.279390 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:06:46.279368 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 23:06:46.283652 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:06:46.283632 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:06:46.287072 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:06:46.287053 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 23:06:49.177951 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:06:49.177913 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:07:03.889762 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:07:03.889724 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:07:15.692755 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:07:15.692713 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:07:30.582811 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:07:30.582773 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:09:28.985750 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:09:28.985712 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:09:40.386655 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:09:40.386619 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:09:48.880759 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:09:48.880723 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:10:00.588316 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:10:00.588287 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:10:12.085645 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:10:12.085598 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:10:23.372373 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:10:23.372338 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:10:32.178505 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:10:32.178467 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:10:43.390351 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:10:43.390270 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:10:54.394624 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:10:54.394587 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:11:02.683358 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:11:02.683323 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:11:05.507293 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:11:05.507265 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:11:13.682888 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:11:13.682854 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:11:37.579572 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:11:37.579537 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:11:45.083198 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:11:45.083161 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:11:46.305772 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:11:46.305748 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:11:46.309513 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:11:46.309490 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 23:11:46.314600 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:11:46.314578 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:11:46.318352 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:11:46.318336 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 23:11:55.376645 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:11:55.376605 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:12:04.979585 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:12:04.979551 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:12:14.386622 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:12:14.386528 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:12:27.179932 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:12:27.179901 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:12:40.979574 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:12:40.979536 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:12:50.187523 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:12:50.187482 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:13:04.186517 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:13:04.186486 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:13:04.221975 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:13:04.221949 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:13:04.309583 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:13:04.309553 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:13:13.174200 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:13:13.174161 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:13:26.283132 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:13:26.283096 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:13:36.286249 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:13:36.286217 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:13:42.253267 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:13:42.253185 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:13:42.353679 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:13:42.353644 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:13:45.583974 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:13:45.583933 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:13:46.064328 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:13:46.064298 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:13:55.181655 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:13:55.181615 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:13:57.388657 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:13:57.388627 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:14:09.079133 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:14:09.079100 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:14:16.976197 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:14:16.976164 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:14:29.280786 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:14:29.280752 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:14:41.276195 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:14:41.276160 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:14:55.182167 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:14:55.182130 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:14:57.347556 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:14:57.347525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:05.034104 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:05.034049 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:08.378134 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:08.378034 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:15:13.274508 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:13.274470 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:17.575698 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:17.575651 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:15:18.344755 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:18.344728 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:21.195217 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:21.195184 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:21.295747 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:21.295713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:21.629918 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:21.629885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:27.230308 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:27.230275 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:27.333802 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:27.333746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:27.435251 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:27.435224 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:27.482189 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:27.482106 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:15:27.534261 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:27.534233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:28.532512 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:28.532484 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:28.596511 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:28.596478 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:28.667870 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:28.667840 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:28.768062 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:28.768031 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:29.181950 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:29.181918 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:29.285608 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:29.285564 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:31.021254 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:31.021230 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:33.614614 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:33.614578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:33.789536 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:33.789509 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:33.888467 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:33.888386 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:34.113551 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:34.113520 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:36.773216 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:36.773183 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:39.468300 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:39.468266 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:39.568438 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:39.568385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:39.706614 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:39.706584 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:39.813284 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:39.813255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:39.916751 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:39.916677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:39.981756 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:39.981717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:41.211057 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:41.211016 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:41.313651 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:41.313592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:41.625260 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:41.625205 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:41.775093 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:41.775038 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:41.876312 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:41.876233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:43.031585 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:43.031533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:45.593505 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:45.593468 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:15:45.670052 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:45.670018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:45.857991 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:45.857912 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:45.961130 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:45.961077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:46.057964 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:46.057934 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:47.627198 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:47.627166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:51.740904 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:51.740866 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:51.845894 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:51.845858 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:51.941182 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:51.941151 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:52.041397 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:52.041371 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:53.134827 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:53.134793 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:53.231906 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:53.231867 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:53.335396 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:53.335365 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:53.752470 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:53.752390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:53.853814 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:53.853782 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:55.362012 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:55.361984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:55.483272 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:55.483241 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:15:58.119537 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:58.119507 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:15:58.515838 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:15:58.515810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:04.179422 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:04.179384 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:04.279341 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:04.279312 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:07.261607 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:07.261580 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:09.866980 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:09.866948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:09.930389 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:09.930359 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:10.077835 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:10.077803 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:10.139449 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:10.139374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:11.230233 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:11.230198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:11.304465 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:11.304431 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:11.363792 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:11.363763 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:11.683843 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:11.683806 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:11.864745 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:11.864711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:11.965400 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:11.965321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:13.408042 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:13.407982 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:14.385747 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:14.385714 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:16:15.655056 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:15.655027 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:15.841134 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:15.841034 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:15.939238 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:15.939158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:16.053990 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:16.053957 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:17.893542 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:17.893504 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:21.725726 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:21.725696 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:21.826954 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:21.826921 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:22.158279 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:22.158253 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:22.258427 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:22.258395 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:22.360263 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:22.360233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:23.686685 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:23.686656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:25.983898 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:25.983866 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:16:29.333198 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:29.333161 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:29.399616 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:29.399582 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:29.485623 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:29.485590 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:29.588069 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:29.587992 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:30.008963 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:30.008933 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:16:46.338204 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:46.338173 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:16:46.342112 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:46.342068 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 23:16:46.346226 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:46.346206 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:16:46.349912 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:46.349896 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 23:16:55.953452 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:16:55.953410 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:01.838747 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:01.838704 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:01.895561 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:01.895534 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:01.957991 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:01.957958 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:02.057886 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:02.057857 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:17.900438 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:17.900400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:23.838930 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:23.838888 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:23.946234 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:23.946204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:24.075634 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:24.075610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:34.573889 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:34.573857 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:35.340193 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:35.340164 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:41.044823 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:41.044771 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:41.101647 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:41.101618 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:41.169980 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:41.169954 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:41.275661 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:41.275617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:41.409769 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:41.409697 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:42.062022 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:42.061989 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:42.163715 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:42.163684 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:42.264541 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:42.264517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:43.781443 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:43.781410 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:45.686586 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:45.686551 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:17:49.477162 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:49.477129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:49.618685 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:49.618655 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:50.134105 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:50.134057 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:56.092817 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:56.092769 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:56.154659 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:56.154631 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:56.310839 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:56.310808 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:56.612045 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:56.612005 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:56.942945 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:56.942860 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:57.475435 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:57.475403 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:17:59.511419 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:59.511375 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:59.598520 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:59.598488 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:59.781019 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:59.780931 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:17:59.944590 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:17:59.944559 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:00.279232 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:00.279199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:00.380386 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:00.380353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:04.003608 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:04.003579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:04.641984 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:04.641955 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:04.740982 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:04.740955 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:05.062579 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:05.062550 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:10.486569 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:10.486484 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:18:10.685433 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:10.685401 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:10.785518 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:10.785489 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:11.064850 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:11.064773 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:11.467178 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:11.467065 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:11.566433 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:11.566404 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:17.031540 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:17.031502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:17.158957 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:17.158925 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:17.294823 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:17.294745 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:17.656745 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:17.656655 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:17.745477 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:17.745448 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:17.857256 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:17.857222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:20.577624 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:20.577586 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:18:23.461170 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:23.461133 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:23.567560 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:23.567525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:23.662340 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:23.662303 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:23.798198 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:23.798149 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:29.395765 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:29.395734 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:29.571849 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:29.571821 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:36.102102 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:36.102051 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:36.421842 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:36.421764 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:36.625871 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:36.625844 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:39.777292 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:39.777256 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:39.865931 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:39.865898 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:39.953705 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:39.953673 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:40.297629 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:40.297589 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:40.400243 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:40.400214 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:40.500705 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:40.500670 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:44.249466 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:44.249431 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:44.514984 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:44.514899 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:44.615189 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:44.615156 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:46.179537 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:46.179504 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:18:55.308759 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:55.308730 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:18:55.371197 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:18:55.371153 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:19:00.982313 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:19:00.982273 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:19:01.167093 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:19:01.167039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:19:01.278940 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:19:01.278909 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:19:21.587539 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:19:21.587500 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:19:26.802215 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:19:26.802167 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:19:27.235506 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:19:27.235434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:19:32.381014 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:19:32.380984 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:19:32.882260 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:19:32.882231 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:19:37.903413 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:19:37.903331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:19:42.200054 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:19:42.200013 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:20:14.241155 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:14.241122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:14.682549 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:14.682517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:17.883010 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:17.882977 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:20:20.270480 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:20.270448 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:25.996038 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:25.995998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:26.096288 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:26.096232 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:26.262498 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:26.262470 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:27.537924 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:27.537884 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:27.629616 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:27.629586 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:27.726888 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:27.726841 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:28.433874 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:28.433843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:28.542319 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:28.542288 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:28.632732 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:28.632701 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:28.787445 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:28.787409 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:20:29.963125 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:29.963076 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:35.486044 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:35.486008 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:35.547441 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:35.547416 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:36.157579 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:36.157539 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:41.724914 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:41.724876 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:41.903131 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:41.903097 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:41.987586 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:41.987517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:41.994007 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:41.993980 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:20:52.431508 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:52.431476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:20:52.788142 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:20:52.788110 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:13.704535 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:13.704454 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:19.407222 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:19.407192 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:19.478529 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:19.478501 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:19.727072 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:19.726996 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:20.033458 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:20.033428 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:20.277431 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:20.277400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:22.646597 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:22.646555 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:22.743141 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:22.743104 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:22.874599 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:22.874566 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:23.272359 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:23.272298 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:23.375836 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:23.375807 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:27.135963 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:27.135923 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:27.456571 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:27.456477 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:27.751008 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:27.750977 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:28.076476 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:28.076380 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:30.475869 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:30.475836 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:21:30.672047 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:30.671818 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:30.754176 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:30.754148 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:30.817209 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:30.817181 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:31.138179 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:31.138102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:31.419634 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:31.419552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:31.519549 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:31.519521 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:35.180196 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:35.180165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:44.651889 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:44.651854 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:44.752558 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:44.752529 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:45.173587 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:45.173557 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:21:46.368294 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:46.368261 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:21:46.371995 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:46.371974 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 23:21:46.377921 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:46.377904 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:21:46.381303 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:46.381280 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 23:21:51.642121 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:51.642066 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:51.741167 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:51.741128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:52.113196 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:52.113163 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:21:52.216228 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:21:52.216196 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:22:03.579605 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.579567 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn"] Jul 08 23:22:03.582917 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.582896 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.586613 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.586592 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0a82e7\"/\"default-dockercfg-bbqz8\"" Jul 08 23:22:03.586613 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.586606 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0a82e7\"/\"openshift-service-ca.crt\"" Jul 08 23:22:03.586766 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.586637 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0a82e7\"/\"kube-root-ca.crt\"" Jul 08 23:22:03.586766 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.586678 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0a82e7\"/\"test-model-0a82e7-kserve-self-signed-certs\"" Jul 08 23:22:03.593027 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.593008 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn"] Jul 08 23:22:03.603540 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.603520 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:22:03.716281 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.716246 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-home\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.716281 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.716286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-model-cache\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.716463 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.716324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-dshm\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.716463 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.716343 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-tmp-dir\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.716463 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.716376 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-tls-certs\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.716570 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.716461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvlk2\" (UniqueName: \"kubernetes.io/projected/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-kube-api-access-zvlk2\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.817479 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.817445 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-dshm\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.817479 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.817480 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-tmp-dir\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.817726 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.817502 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-tls-certs\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.817726 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.817555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvlk2\" (UniqueName: \"kubernetes.io/projected/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-kube-api-access-zvlk2\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.817726 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.817600 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-home\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.817726 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.817639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-model-cache\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.817923 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.817874 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-tmp-dir\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.818009 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.817988 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-home\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.818051 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.818031 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-model-cache\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.819838 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.819817 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-dshm\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.819980 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.819963 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-tls-certs\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.825601 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.825575 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvlk2\" (UniqueName: \"kubernetes.io/projected/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-kube-api-access-zvlk2\") pod \"test-model-0a82e7-kserve-5dd89b46c5-nc6bn\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:03.893262 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:03.893156 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:04.027486 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:04.027455 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn"] Jul 08 23:22:04.030628 ip-10-0-143-210 kubenswrapper[2564]: W0708 23:22:04.030601 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb76ffa9a_c88d_4b2a_8784_9a99b4e9a205.slice/crio-cad9e125cf7b44878d93a25630ee344a1b1c98fc151e5cb630e22675dd9a4192 WatchSource:0}: Error finding container cad9e125cf7b44878d93a25630ee344a1b1c98fc151e5cb630e22675dd9a4192: Status 404 returned error can't find the container with id cad9e125cf7b44878d93a25630ee344a1b1c98fc151e5cb630e22675dd9a4192 Jul 08 23:22:04.062825 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:04.062793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" event={"ID":"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205","Type":"ContainerStarted","Data":"cad9e125cf7b44878d93a25630ee344a1b1c98fc151e5cb630e22675dd9a4192"} Jul 08 23:22:05.069677 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:05.069638 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" event={"ID":"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205","Type":"ContainerStarted","Data":"13e6664fd852b5de36bcda4ce9b4a67a0e09c134d02b7bf5b578c7722f5cf42f"} Jul 08 23:22:05.086457 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:05.086410 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" podStartSLOduration=2.0863963229999998 podStartE2EDuration="2.086396323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:22:05.085384944 +0000 UTC m=+2119.340832648" watchObservedRunningTime="2026-07-08 23:22:05.086396323 +0000 UTC m=+2119.341844026" Jul 08 23:22:06.873847 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:06.873764 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:22:13.893651 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:13.893610 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:13.894053 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:13.893866 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:13.906701 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:13.906672 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:14.125948 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:14.125923 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:22:21.581043 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:21.581005 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:22:27.678303 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.678271 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn"] Jul 08 23:22:27.681598 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.681581 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.684219 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.684200 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a78f59\"/\"kube-root-ca.crt\"" Jul 08 23:22:27.685600 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.685581 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a78f59\"/\"test-model-a78f59-kserve-self-signed-certs\"" Jul 08 23:22:27.685700 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.685584 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a78f59\"/\"openshift-service-ca.crt\"" Jul 08 23:22:27.685700 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.685585 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a78f59\"/\"default-dockercfg-kkj9w\"" Jul 08 23:22:27.692237 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.692217 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn"] Jul 08 23:22:27.702573 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.702552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:22:27.711438 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.711416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-dshm\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.711535 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.711454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54q27\" (UniqueName: \"kubernetes.io/projected/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-kube-api-access-54q27\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.711535 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.711485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-model-cache\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.711535 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.711509 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-tmp-dir\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.711535 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.711532 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-tls-certs\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.711692 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.711564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-home\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.812982 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.812949 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-dshm\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.813186 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.813003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54q27\" (UniqueName: \"kubernetes.io/projected/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-kube-api-access-54q27\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.813186 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.813034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-model-cache\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.813186 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.813067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-tmp-dir\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.813186 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.813126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-tls-certs\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.813186 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.813173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-home\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.813496 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.813458 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-model-cache\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.813496 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.813473 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-tmp-dir\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.813601 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.813559 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-home\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.815321 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.815301 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-dshm\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.815688 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.815671 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-tls-certs\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.821524 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.821496 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54q27\" (UniqueName: \"kubernetes.io/projected/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-kube-api-access-54q27\") pod \"test-model-a78f59-kserve-7898858d5-2t9xn\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:27.991243 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:27.991172 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:28.123792 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:28.123764 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn"] Jul 08 23:22:28.126231 ip-10-0-143-210 kubenswrapper[2564]: W0708 23:22:28.126201 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c796adf_eaf5_45a1_b9f3_01119e4a3aaf.slice/crio-7836f914a8b46400a633ebdc75100e39e30ca93e1c3967c2cab9750aa1191699 WatchSource:0}: Error finding container 7836f914a8b46400a633ebdc75100e39e30ca93e1c3967c2cab9750aa1191699: Status 404 returned error can't find the container with id 7836f914a8b46400a633ebdc75100e39e30ca93e1c3967c2cab9750aa1191699 Jul 08 23:22:28.188251 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:28.188225 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" event={"ID":"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf","Type":"ContainerStarted","Data":"347e147c89ef9c015bc092ebd57395a6f4b16f991ec99d116da164db5fcad35f"} Jul 08 23:22:28.188367 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:28.188265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" event={"ID":"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf","Type":"ContainerStarted","Data":"7836f914a8b46400a633ebdc75100e39e30ca93e1c3967c2cab9750aa1191699"} Jul 08 23:22:28.205053 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:28.205010 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" podStartSLOduration=1.204992658 podStartE2EDuration="1.204992658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:22:28.203202105 +0000 UTC m=+2142.458649831" watchObservedRunningTime="2026-07-08 23:22:28.204992658 +0000 UTC m=+2142.460440365" Jul 08 23:22:32.583732 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:32.583703 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:22:37.991816 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:37.991726 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:37.991816 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:37.991778 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:38.003908 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:38.003886 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:38.247272 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:38.247249 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:22:44.757787 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:44.757751 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:22:49.761415 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:49.761385 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn"] Jul 08 23:22:49.780437 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:49.780412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:22:50.693824 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:50.693794 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn"] Jul 08 23:22:50.711568 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:50.711540 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:22:50.856272 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:50.856242 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:22:51.253270 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:51.253239 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:22:51.366878 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:51.366846 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:22:51.467775 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:51.467741 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:22:51.609399 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:51.609320 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:22:56.331459 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:56.331422 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:22:56.886036 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:56.886003 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:22:57.428427 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:57.428396 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:22:57.529886 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:22:57.529856 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:04.782017 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:04.781958 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" podUID="b76ffa9a-c88d-4b2a-8784-9a99b4e9a205" containerName="main" containerID="cri-o://13e6664fd852b5de36bcda4ce9b4a67a0e09c134d02b7bf5b578c7722f5cf42f" gracePeriod=45 Jul 08 23:23:05.029314 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.029292 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:23:05.137893 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.137801 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-tls-certs\") pod \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " Jul 08 23:23:05.137893 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.137857 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-tmp-dir\") pod \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " Jul 08 23:23:05.138172 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.137930 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-dshm\") pod \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " Jul 08 23:23:05.138172 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.137971 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-home\") pod \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " Jul 08 23:23:05.138172 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.138047 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-model-cache\") pod \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " Jul 08 23:23:05.138172 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.138073 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvlk2\" (UniqueName: \"kubernetes.io/projected/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-kube-api-access-zvlk2\") pod \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\" (UID: \"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205\") " Jul 08 23:23:05.138371 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.138219 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b76ffa9a-c88d-4b2a-8784-9a99b4e9a205" (UID: "b76ffa9a-c88d-4b2a-8784-9a99b4e9a205"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:23:05.138371 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.138335 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-model-cache" (OuterVolumeSpecName: "model-cache") pod "b76ffa9a-c88d-4b2a-8784-9a99b4e9a205" (UID: "b76ffa9a-c88d-4b2a-8784-9a99b4e9a205"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:23:05.138371 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.138340 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-home" (OuterVolumeSpecName: "home") pod "b76ffa9a-c88d-4b2a-8784-9a99b4e9a205" (UID: "b76ffa9a-c88d-4b2a-8784-9a99b4e9a205"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:23:05.138495 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.138458 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-tmp-dir\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:23:05.138495 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.138478 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-home\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:23:05.138495 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.138491 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-model-cache\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:23:05.140179 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.140152 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-dshm" (OuterVolumeSpecName: "dshm") pod "b76ffa9a-c88d-4b2a-8784-9a99b4e9a205" (UID: "b76ffa9a-c88d-4b2a-8784-9a99b4e9a205"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:23:05.140284 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.140228 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b76ffa9a-c88d-4b2a-8784-9a99b4e9a205" (UID: "b76ffa9a-c88d-4b2a-8784-9a99b4e9a205"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 23:23:05.140284 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.140233 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-kube-api-access-zvlk2" (OuterVolumeSpecName: "kube-api-access-zvlk2") pod "b76ffa9a-c88d-4b2a-8784-9a99b4e9a205" (UID: "b76ffa9a-c88d-4b2a-8784-9a99b4e9a205"). InnerVolumeSpecName "kube-api-access-zvlk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 23:23:05.239869 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.239841 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-dshm\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:23:05.239869 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.239865 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvlk2\" (UniqueName: \"kubernetes.io/projected/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-kube-api-access-zvlk2\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:23:05.240061 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.239877 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205-tls-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:23:05.385335 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.385294 2564 generic.go:356] "Generic (PLEG): container finished" podID="b76ffa9a-c88d-4b2a-8784-9a99b4e9a205" containerID="13e6664fd852b5de36bcda4ce9b4a67a0e09c134d02b7bf5b578c7722f5cf42f" exitCode=0 Jul 08 23:23:05.385506 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.385354 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" Jul 08 23:23:05.385506 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.385383 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" event={"ID":"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205","Type":"ContainerDied","Data":"13e6664fd852b5de36bcda4ce9b4a67a0e09c134d02b7bf5b578c7722f5cf42f"} Jul 08 23:23:05.385506 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.385432 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn" event={"ID":"b76ffa9a-c88d-4b2a-8784-9a99b4e9a205","Type":"ContainerDied","Data":"cad9e125cf7b44878d93a25630ee344a1b1c98fc151e5cb630e22675dd9a4192"} Jul 08 23:23:05.385506 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.385450 2564 scope.go:117] "RemoveContainer" containerID="13e6664fd852b5de36bcda4ce9b4a67a0e09c134d02b7bf5b578c7722f5cf42f" Jul 08 23:23:05.394611 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.394593 2564 scope.go:117] "RemoveContainer" containerID="13e6664fd852b5de36bcda4ce9b4a67a0e09c134d02b7bf5b578c7722f5cf42f" Jul 08 23:23:05.394882 ip-10-0-143-210 kubenswrapper[2564]: E0708 23:23:05.394859 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e6664fd852b5de36bcda4ce9b4a67a0e09c134d02b7bf5b578c7722f5cf42f\": container with ID starting with 13e6664fd852b5de36bcda4ce9b4a67a0e09c134d02b7bf5b578c7722f5cf42f not found: ID does not exist" containerID="13e6664fd852b5de36bcda4ce9b4a67a0e09c134d02b7bf5b578c7722f5cf42f" Jul 08 23:23:05.394966 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.394890 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e6664fd852b5de36bcda4ce9b4a67a0e09c134d02b7bf5b578c7722f5cf42f"} err="failed to get container status \"13e6664fd852b5de36bcda4ce9b4a67a0e09c134d02b7bf5b578c7722f5cf42f\": rpc error: code = NotFound desc = could not find container \"13e6664fd852b5de36bcda4ce9b4a67a0e09c134d02b7bf5b578c7722f5cf42f\": container with ID starting with 13e6664fd852b5de36bcda4ce9b4a67a0e09c134d02b7bf5b578c7722f5cf42f not found: ID does not exist" Jul 08 23:23:05.405612 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.405585 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn"] Jul 08 23:23:05.411178 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.411159 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0a82e7/test-model-0a82e7-kserve-5dd89b46c5-nc6bn"] Jul 08 23:23:05.714501 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.714409 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" podUID="1c796adf-eaf5-45a1-b9f3-01119e4a3aaf" containerName="main" containerID="cri-o://347e147c89ef9c015bc092ebd57395a6f4b16f991ec99d116da164db5fcad35f" gracePeriod=45 Jul 08 23:23:05.957297 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:05.957275 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:23:06.146537 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.146508 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-tmp-dir\") pod \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " Jul 08 23:23:06.146705 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.146547 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-tls-certs\") pod \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " Jul 08 23:23:06.146705 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.146586 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54q27\" (UniqueName: \"kubernetes.io/projected/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-kube-api-access-54q27\") pod \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " Jul 08 23:23:06.146705 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.146634 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-dshm\") pod \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " Jul 08 23:23:06.146705 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.146690 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-model-cache\") pod \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " Jul 08 23:23:06.146905 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.146753 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-home\") pod \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\" (UID: \"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf\") " Jul 08 23:23:06.146905 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.146795 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c796adf-eaf5-45a1-b9f3-01119e4a3aaf" (UID: "1c796adf-eaf5-45a1-b9f3-01119e4a3aaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:23:06.147021 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.146962 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c796adf-eaf5-45a1-b9f3-01119e4a3aaf" (UID: "1c796adf-eaf5-45a1-b9f3-01119e4a3aaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:23:06.147123 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.147096 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-home" (OuterVolumeSpecName: "home") pod "1c796adf-eaf5-45a1-b9f3-01119e4a3aaf" (UID: "1c796adf-eaf5-45a1-b9f3-01119e4a3aaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:23:06.147185 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.147151 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-model-cache\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:23:06.147229 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.147190 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-tmp-dir\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:23:06.148831 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.148804 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-kube-api-access-54q27" (OuterVolumeSpecName: "kube-api-access-54q27") pod "1c796adf-eaf5-45a1-b9f3-01119e4a3aaf" (UID: "1c796adf-eaf5-45a1-b9f3-01119e4a3aaf"). InnerVolumeSpecName "kube-api-access-54q27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 23:23:06.149122 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.149075 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c796adf-eaf5-45a1-b9f3-01119e4a3aaf" (UID: "1c796adf-eaf5-45a1-b9f3-01119e4a3aaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 23:23:06.149122 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.149100 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-dshm" (OuterVolumeSpecName: "dshm") pod "1c796adf-eaf5-45a1-b9f3-01119e4a3aaf" (UID: "1c796adf-eaf5-45a1-b9f3-01119e4a3aaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:23:06.248522 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.248495 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-home\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:23:06.248522 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.248518 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-tls-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:23:06.248651 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.248528 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54q27\" (UniqueName: \"kubernetes.io/projected/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-kube-api-access-54q27\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:23:06.248651 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.248537 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf-dshm\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:23:06.268166 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.268142 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b76ffa9a-c88d-4b2a-8784-9a99b4e9a205" path="/var/lib/kubelet/pods/b76ffa9a-c88d-4b2a-8784-9a99b4e9a205/volumes" Jul 08 23:23:06.391713 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.391686 2564 generic.go:356] "Generic (PLEG): container finished" podID="1c796adf-eaf5-45a1-b9f3-01119e4a3aaf" containerID="347e147c89ef9c015bc092ebd57395a6f4b16f991ec99d116da164db5fcad35f" exitCode=0 Jul 08 23:23:06.391814 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.391730 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" Jul 08 23:23:06.391814 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.391727 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" event={"ID":"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf","Type":"ContainerDied","Data":"347e147c89ef9c015bc092ebd57395a6f4b16f991ec99d116da164db5fcad35f"} Jul 08 23:23:06.391814 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.391763 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn" event={"ID":"1c796adf-eaf5-45a1-b9f3-01119e4a3aaf","Type":"ContainerDied","Data":"7836f914a8b46400a633ebdc75100e39e30ca93e1c3967c2cab9750aa1191699"} Jul 08 23:23:06.391814 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.391780 2564 scope.go:117] "RemoveContainer" containerID="347e147c89ef9c015bc092ebd57395a6f4b16f991ec99d116da164db5fcad35f" Jul 08 23:23:06.400053 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.400007 2564 scope.go:117] "RemoveContainer" containerID="347e147c89ef9c015bc092ebd57395a6f4b16f991ec99d116da164db5fcad35f" Jul 08 23:23:06.400333 ip-10-0-143-210 kubenswrapper[2564]: E0708 23:23:06.400315 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347e147c89ef9c015bc092ebd57395a6f4b16f991ec99d116da164db5fcad35f\": container with ID starting with 347e147c89ef9c015bc092ebd57395a6f4b16f991ec99d116da164db5fcad35f not found: ID does not exist" containerID="347e147c89ef9c015bc092ebd57395a6f4b16f991ec99d116da164db5fcad35f" Jul 08 23:23:06.400392 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.400341 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347e147c89ef9c015bc092ebd57395a6f4b16f991ec99d116da164db5fcad35f"} err="failed to get container status \"347e147c89ef9c015bc092ebd57395a6f4b16f991ec99d116da164db5fcad35f\": rpc error: code = NotFound desc = could not find container \"347e147c89ef9c015bc092ebd57395a6f4b16f991ec99d116da164db5fcad35f\": container with ID starting with 347e147c89ef9c015bc092ebd57395a6f4b16f991ec99d116da164db5fcad35f not found: ID does not exist" Jul 08 23:23:06.409041 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.409019 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn"] Jul 08 23:23:06.411658 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:06.411640 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a78f59/test-model-a78f59-kserve-7898858d5-2t9xn"] Jul 08 23:23:08.268070 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:08.268041 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c796adf-eaf5-45a1-b9f3-01119e4a3aaf" path="/var/lib/kubelet/pods/1c796adf-eaf5-45a1-b9f3-01119e4a3aaf/volumes" Jul 08 23:23:32.849456 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:32.849423 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:33.097201 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:33.097167 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:33.487589 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:33.487562 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:33.883776 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:33.883744 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:33.981938 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:33.981911 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:34.033598 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:34.033570 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:34.170267 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:34.170194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:36.476818 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:36.476789 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:23:39.730789 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:39.730757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:39.831024 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:39.830985 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:39.986245 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:39.986162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:40.545203 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:40.545158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:40.843973 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:40.843905 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:41.070936 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:41.070904 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:43.462174 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:43.462139 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:43.561535 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:43.561502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:43.686954 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:43.686923 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:43.808682 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:43.808640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:44.190352 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:44.190274 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:44.303355 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:44.303331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:44.395634 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:44.395592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:45.635958 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:45.635930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:45.734036 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:45.734008 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:46.492869 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:46.492835 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:46.593122 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:46.593069 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:47.927254 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:47.927222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:48.378363 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:48.378331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:48.669499 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:48.669389 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:49.240948 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:49.240922 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:51.378353 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:51.378316 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:23:51.472987 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:51.472959 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:51.577994 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:51.577966 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:51.676704 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:51.676611 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:51.772765 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:51.772727 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:52.226077 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:52.226039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:52.326410 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:52.326371 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:23:55.951466 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:23:55.951423 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:03.956118 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:03.956069 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:04.313210 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:04.313181 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:04.528932 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:04.528901 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:04.633039 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:04.632954 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:04.807132 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:04.807073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:08.182069 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:08.181989 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:24:09.945251 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:09.945219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:10.039486 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:10.039453 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:10.220596 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:10.220514 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:10.575815 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:10.575781 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:10.969969 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:10.969885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:11.068493 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:11.068449 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:11.169095 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:11.169067 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:11.297867 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:11.297840 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:16.612303 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:16.612276 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:16.674070 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:16.674047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:16.865760 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:16.865671 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:17.390400 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:17.390336 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:17.674840 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:17.674744 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:17.918870 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:17.918824 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:20.686663 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:20.686636 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:20.771607 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:20.771575 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:20.842715 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:20.842684 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:21.312940 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:21.312908 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:21.450156 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:21.450123 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:21.551295 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:21.551262 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:22.486484 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:22.486449 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:24:25.178304 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:25.178268 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:25.511050 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:25.511015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:25.803021 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:25.802908 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:26.170995 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:26.170907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:28.806376 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:28.806341 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:28.906077 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:28.906045 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:28.968641 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:28.968607 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:29.420245 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:29.420215 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:29.602598 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:29.602549 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:29.705477 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:29.705374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:33.273923 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:33.273886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:33.483435 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:33.483400 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:24:42.489143 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:42.489111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:42.590419 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:42.590383 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:50.282914 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:50.282872 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:50.385783 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:50.385746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:58.485233 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.485197 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj"] Jul 08 23:24:58.485723 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.485666 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c796adf-eaf5-45a1-b9f3-01119e4a3aaf" containerName="main" Jul 08 23:24:58.485723 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.485681 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c796adf-eaf5-45a1-b9f3-01119e4a3aaf" containerName="main" Jul 08 23:24:58.485723 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.485693 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76ffa9a-c88d-4b2a-8784-9a99b4e9a205" containerName="main" Jul 08 23:24:58.485723 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.485698 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76ffa9a-c88d-4b2a-8784-9a99b4e9a205" containerName="main" Jul 08 23:24:58.485963 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.485760 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76ffa9a-c88d-4b2a-8784-9a99b4e9a205" containerName="main" Jul 08 23:24:58.485963 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.485772 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c796adf-eaf5-45a1-b9f3-01119e4a3aaf" containerName="main" Jul 08 23:24:58.490474 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.490458 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.494466 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.494290 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-77dfb5\"/\"auth-test-model-77dfb5-kserve-self-signed-certs\"" Jul 08 23:24:58.494786 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.494643 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-77dfb5\"/\"openshift-service-ca.crt\"" Jul 08 23:24:58.494786 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.494682 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-77dfb5\"/\"default-dockercfg-sn2xr\"" Jul 08 23:24:58.494977 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.494854 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-77dfb5\"/\"kube-root-ca.crt\"" Jul 08 23:24:58.496939 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.496917 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj"] Jul 08 23:24:58.509449 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.509420 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:24:58.614002 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.613971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-dshm\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.614002 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.614007 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-tmp-dir\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.614240 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.614030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sldg5\" (UniqueName: \"kubernetes.io/projected/6e18472a-a777-49e3-b27b-6dc7a70ba516-kube-api-access-sldg5\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.614240 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.614161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-home\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.614240 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.614188 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-model-cache\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.614240 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.614208 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e18472a-a777-49e3-b27b-6dc7a70ba516-tls-certs\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.715048 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.715009 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-home\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.715048 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.715049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-model-cache\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.715295 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.715067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e18472a-a777-49e3-b27b-6dc7a70ba516-tls-certs\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.715295 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.715148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-dshm\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.715295 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.715173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-tmp-dir\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.715295 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.715206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sldg5\" (UniqueName: \"kubernetes.io/projected/6e18472a-a777-49e3-b27b-6dc7a70ba516-kube-api-access-sldg5\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.715526 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.715489 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-model-cache\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.715613 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.715536 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-home\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.715613 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.715576 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-tmp-dir\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.717404 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.717376 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-dshm\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.717669 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.717651 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e18472a-a777-49e3-b27b-6dc7a70ba516-tls-certs\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.723332 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.723310 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sldg5\" (UniqueName: \"kubernetes.io/projected/6e18472a-a777-49e3-b27b-6dc7a70ba516-kube-api-access-sldg5\") pod \"auth-test-model-77dfb5-kserve-c66c6b47-qqkpj\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.801290 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.801257 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:24:58.933747 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:58.933714 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj"] Jul 08 23:24:58.937139 ip-10-0-143-210 kubenswrapper[2564]: W0708 23:24:58.937107 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e18472a_a777_49e3_b27b_6dc7a70ba516.slice/crio-7a406e75be0458625d14455c0250b5bc8858e6eee271433cb68392de27a6cda9 WatchSource:0}: Error finding container 7a406e75be0458625d14455c0250b5bc8858e6eee271433cb68392de27a6cda9: Status 404 returned error can't find the container with id 7a406e75be0458625d14455c0250b5bc8858e6eee271433cb68392de27a6cda9 Jul 08 23:24:59.947726 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:59.947689 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" event={"ID":"6e18472a-a777-49e3-b27b-6dc7a70ba516","Type":"ContainerStarted","Data":"a9dfed89d5fc9070d60d0aed92a871dce1319ef5757340411f2f87d21a146a58"} Jul 08 23:24:59.948250 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:59.947734 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" event={"ID":"6e18472a-a777-49e3-b27b-6dc7a70ba516","Type":"ContainerStarted","Data":"7a406e75be0458625d14455c0250b5bc8858e6eee271433cb68392de27a6cda9"} Jul 08 23:24:59.965382 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:24:59.965333 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" podStartSLOduration=1.965318505 podStartE2EDuration="1.965318505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:24:59.964318249 +0000 UTC m=+2294.219765953" watchObservedRunningTime="2026-07-08 23:24:59.965318505 +0000 UTC m=+2294.220766186" Jul 08 23:25:01.685134 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:25:01.685095 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:25:08.801430 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:25:08.801389 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:25:08.801801 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:25:08.801451 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:25:08.814096 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:25:08.814060 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:25:08.999279 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:25:08.999250 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:25:25.817623 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:25:25.817588 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:25:29.889327 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:25:29.889293 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:25:47.576610 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:25:47.576411 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:26:06.582861 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:26:06.582833 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:26:24.085829 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:26:24.085793 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:26:42.077223 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:26:42.077189 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:26:46.408979 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:26:46.408945 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:26:46.413128 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:26:46.413103 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 23:26:46.417902 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:26:46.417883 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:26:46.421280 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:26:46.421264 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 23:26:56.378457 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:26:56.378415 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:27:15.278963 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:15.278879 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:27:28.961629 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:28.961578 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj"] Jul 08 23:27:29.022226 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:29.022175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:27:30.447346 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:30.447316 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:27:34.182058 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:34.182020 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:27:43.980994 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:43.980937 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" podUID="6e18472a-a777-49e3-b27b-6dc7a70ba516" containerName="main" containerID="cri-o://a9dfed89d5fc9070d60d0aed92a871dce1319ef5757340411f2f87d21a146a58" gracePeriod=45 Jul 08 23:27:44.184897 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.184867 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m"] Jul 08 23:27:44.199799 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.199728 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.200677 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.200655 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m"] Jul 08 23:27:44.202293 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.202229 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-77dfb5\"/\"sub-test-model-77dfb5-kserve-self-signed-certs\"" Jul 08 23:27:44.217965 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.217938 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:27:44.239385 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.239341 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:27:44.275905 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.275881 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-dshm\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.276008 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.275942 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-tls-certs\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.276074 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.276039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-tmp-dir\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.276074 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.276061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2sbb\" (UniqueName: \"kubernetes.io/projected/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-kube-api-access-t2sbb\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.276172 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.276141 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-home\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.276209 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.276196 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-model-cache\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.376877 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.376853 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e18472a-a777-49e3-b27b-6dc7a70ba516-tls-certs\") pod \"6e18472a-a777-49e3-b27b-6dc7a70ba516\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " Jul 08 23:27:44.377000 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.376883 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-home\") pod \"6e18472a-a777-49e3-b27b-6dc7a70ba516\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " Jul 08 23:27:44.377000 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.376902 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-model-cache\") pod \"6e18472a-a777-49e3-b27b-6dc7a70ba516\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " Jul 08 23:27:44.377000 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.376930 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sldg5\" (UniqueName: \"kubernetes.io/projected/6e18472a-a777-49e3-b27b-6dc7a70ba516-kube-api-access-sldg5\") pod \"6e18472a-a777-49e3-b27b-6dc7a70ba516\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " Jul 08 23:27:44.377000 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.376956 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-tmp-dir\") pod \"6e18472a-a777-49e3-b27b-6dc7a70ba516\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " Jul 08 23:27:44.377000 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.376978 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-dshm\") pod \"6e18472a-a777-49e3-b27b-6dc7a70ba516\" (UID: \"6e18472a-a777-49e3-b27b-6dc7a70ba516\") " Jul 08 23:27:44.377284 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-tmp-dir\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.377284 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377106 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2sbb\" (UniqueName: \"kubernetes.io/projected/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-kube-api-access-t2sbb\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.377284 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-home\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.377284 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-model-cache\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.377284 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-dshm\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.377284 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377210 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-home" (OuterVolumeSpecName: "home") pod "6e18472a-a777-49e3-b27b-6dc7a70ba516" (UID: "6e18472a-a777-49e3-b27b-6dc7a70ba516"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:27:44.377284 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377226 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e18472a-a777-49e3-b27b-6dc7a70ba516" (UID: "6e18472a-a777-49e3-b27b-6dc7a70ba516"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:27:44.377631 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377311 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e18472a-a777-49e3-b27b-6dc7a70ba516" (UID: "6e18472a-a777-49e3-b27b-6dc7a70ba516"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:27:44.377631 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377611 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-model-cache\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.377735 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377640 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-tmp-dir\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.377735 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-tls-certs\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.377735 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377732 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-home\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:27:44.377868 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377749 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-model-cache\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:27:44.377868 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377753 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-home\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.377868 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.377835 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-tmp-dir\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:27:44.379356 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.379310 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e18472a-a777-49e3-b27b-6dc7a70ba516-kube-api-access-sldg5" (OuterVolumeSpecName: "kube-api-access-sldg5") pod "6e18472a-a777-49e3-b27b-6dc7a70ba516" (UID: "6e18472a-a777-49e3-b27b-6dc7a70ba516"). InnerVolumeSpecName "kube-api-access-sldg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 23:27:44.379356 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.379308 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-dshm" (OuterVolumeSpecName: "dshm") pod "6e18472a-a777-49e3-b27b-6dc7a70ba516" (UID: "6e18472a-a777-49e3-b27b-6dc7a70ba516"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:27:44.379501 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.379479 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e18472a-a777-49e3-b27b-6dc7a70ba516-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e18472a-a777-49e3-b27b-6dc7a70ba516" (UID: "6e18472a-a777-49e3-b27b-6dc7a70ba516"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 23:27:44.379814 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.379797 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-dshm\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.380248 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.380231 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-tls-certs\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.384637 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.384618 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2sbb\" (UniqueName: \"kubernetes.io/projected/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-kube-api-access-t2sbb\") pod \"sub-test-model-77dfb5-kserve-79855df56f-q7q5m\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.478585 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.478562 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e18472a-a777-49e3-b27b-6dc7a70ba516-tls-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:27:44.478585 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.478584 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sldg5\" (UniqueName: \"kubernetes.io/projected/6e18472a-a777-49e3-b27b-6dc7a70ba516-kube-api-access-sldg5\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:27:44.478747 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.478594 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e18472a-a777-49e3-b27b-6dc7a70ba516-dshm\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:27:44.510918 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.510896 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:44.641696 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.641670 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m"] Jul 08 23:27:44.643488 ip-10-0-143-210 kubenswrapper[2564]: W0708 23:27:44.643463 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod148cb8ac_d0e7_4fd8_9682_462f844cc7b2.slice/crio-750c3592b9bb7aeecc9b6c9e94084ae3b293e63cc11ef45571a8c6ce1297027e WatchSource:0}: Error finding container 750c3592b9bb7aeecc9b6c9e94084ae3b293e63cc11ef45571a8c6ce1297027e: Status 404 returned error can't find the container with id 750c3592b9bb7aeecc9b6c9e94084ae3b293e63cc11ef45571a8c6ce1297027e Jul 08 23:27:44.741487 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.741459 2564 generic.go:356] "Generic (PLEG): container finished" podID="6e18472a-a777-49e3-b27b-6dc7a70ba516" containerID="a9dfed89d5fc9070d60d0aed92a871dce1319ef5757340411f2f87d21a146a58" exitCode=0 Jul 08 23:27:44.741625 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.741502 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" Jul 08 23:27:44.741625 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.741546 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" event={"ID":"6e18472a-a777-49e3-b27b-6dc7a70ba516","Type":"ContainerDied","Data":"a9dfed89d5fc9070d60d0aed92a871dce1319ef5757340411f2f87d21a146a58"} Jul 08 23:27:44.741625 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.741585 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj" event={"ID":"6e18472a-a777-49e3-b27b-6dc7a70ba516","Type":"ContainerDied","Data":"7a406e75be0458625d14455c0250b5bc8858e6eee271433cb68392de27a6cda9"} Jul 08 23:27:44.741625 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.741603 2564 scope.go:117] "RemoveContainer" containerID="a9dfed89d5fc9070d60d0aed92a871dce1319ef5757340411f2f87d21a146a58" Jul 08 23:27:44.743331 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.743307 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" event={"ID":"148cb8ac-d0e7-4fd8-9682-462f844cc7b2","Type":"ContainerStarted","Data":"ef35a4e0410071e009b96c31e13ba7eb8262672db5973b7319c2b0427953bbd7"} Jul 08 23:27:44.743429 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.743345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" event={"ID":"148cb8ac-d0e7-4fd8-9682-462f844cc7b2","Type":"ContainerStarted","Data":"750c3592b9bb7aeecc9b6c9e94084ae3b293e63cc11ef45571a8c6ce1297027e"} Jul 08 23:27:44.751333 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.751316 2564 scope.go:117] "RemoveContainer" containerID="a9dfed89d5fc9070d60d0aed92a871dce1319ef5757340411f2f87d21a146a58" Jul 08 23:27:44.751589 ip-10-0-143-210 kubenswrapper[2564]: E0708 23:27:44.751572 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9dfed89d5fc9070d60d0aed92a871dce1319ef5757340411f2f87d21a146a58\": container with ID starting with a9dfed89d5fc9070d60d0aed92a871dce1319ef5757340411f2f87d21a146a58 not found: ID does not exist" containerID="a9dfed89d5fc9070d60d0aed92a871dce1319ef5757340411f2f87d21a146a58" Jul 08 23:27:44.751641 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.751598 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9dfed89d5fc9070d60d0aed92a871dce1319ef5757340411f2f87d21a146a58"} err="failed to get container status \"a9dfed89d5fc9070d60d0aed92a871dce1319ef5757340411f2f87d21a146a58\": rpc error: code = NotFound desc = could not find container \"a9dfed89d5fc9070d60d0aed92a871dce1319ef5757340411f2f87d21a146a58\": container with ID starting with a9dfed89d5fc9070d60d0aed92a871dce1319ef5757340411f2f87d21a146a58 not found: ID does not exist" Jul 08 23:27:44.760308 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.760270 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" podStartSLOduration=0.760257636 podStartE2EDuration="760.257636ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:27:44.759034891 +0000 UTC m=+2459.014482596" watchObservedRunningTime="2026-07-08 23:27:44.760257636 +0000 UTC m=+2459.015705339" Jul 08 23:27:44.771923 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.771870 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj"] Jul 08 23:27:44.775229 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:44.775211 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-77dfb5/auth-test-model-77dfb5-kserve-c66c6b47-qqkpj"] Jul 08 23:27:46.268992 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:46.268962 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e18472a-a777-49e3-b27b-6dc7a70ba516" path="/var/lib/kubelet/pods/6e18472a-a777-49e3-b27b-6dc7a70ba516/volumes" Jul 08 23:27:49.380268 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:49.380234 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:27:49.437346 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:49.437308 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:27:49.779439 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:49.779407 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:27:54.511770 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:54.511728 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:54.512247 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:54.511811 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:54.525475 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:54.525450 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:27:54.801922 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:27:54.801827 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:28:11.771686 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:28:11.771655 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:28:16.387135 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:28:16.387101 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:28:46.446939 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:28:46.446908 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:28:46.553799 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:28:46.553767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:28:51.379712 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:28:51.379675 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:29:11.176061 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:11.176020 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:29:18.393741 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:18.393703 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m"] Jul 08 23:29:18.411220 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:18.411188 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:29:19.383685 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:19.383651 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:29:26.120492 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:26.120457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:29:31.581428 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:31.581395 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:29:33.415991 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.415937 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" podUID="148cb8ac-d0e7-4fd8-9682-462f844cc7b2" containerName="main" containerID="cri-o://ef35a4e0410071e009b96c31e13ba7eb8262672db5973b7319c2b0427953bbd7" gracePeriod=45 Jul 08 23:29:33.662303 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.662278 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:29:33.752976 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.752954 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-dshm\") pod \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " Jul 08 23:29:33.753163 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.752987 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-tls-certs\") pod \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " Jul 08 23:29:33.753163 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.753011 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2sbb\" (UniqueName: \"kubernetes.io/projected/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-kube-api-access-t2sbb\") pod \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " Jul 08 23:29:33.753163 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.753056 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-tmp-dir\") pod \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " Jul 08 23:29:33.753163 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.753075 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-model-cache\") pod \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " Jul 08 23:29:33.753163 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.753121 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-home\") pod \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\" (UID: \"148cb8ac-d0e7-4fd8-9682-462f844cc7b2\") " Jul 08 23:29:33.753445 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.753370 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "148cb8ac-d0e7-4fd8-9682-462f844cc7b2" (UID: "148cb8ac-d0e7-4fd8-9682-462f844cc7b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:29:33.753504 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.753413 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "148cb8ac-d0e7-4fd8-9682-462f844cc7b2" (UID: "148cb8ac-d0e7-4fd8-9682-462f844cc7b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:29:33.753504 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.753479 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-home" (OuterVolumeSpecName: "home") pod "148cb8ac-d0e7-4fd8-9682-462f844cc7b2" (UID: "148cb8ac-d0e7-4fd8-9682-462f844cc7b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:29:33.755202 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.755176 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-dshm" (OuterVolumeSpecName: "dshm") pod "148cb8ac-d0e7-4fd8-9682-462f844cc7b2" (UID: "148cb8ac-d0e7-4fd8-9682-462f844cc7b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:29:33.755202 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.755187 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "148cb8ac-d0e7-4fd8-9682-462f844cc7b2" (UID: "148cb8ac-d0e7-4fd8-9682-462f844cc7b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 23:29:33.755362 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.755180 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-kube-api-access-t2sbb" (OuterVolumeSpecName: "kube-api-access-t2sbb") pod "148cb8ac-d0e7-4fd8-9682-462f844cc7b2" (UID: "148cb8ac-d0e7-4fd8-9682-462f844cc7b2"). InnerVolumeSpecName "kube-api-access-t2sbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 23:29:33.854514 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.854476 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-dshm\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:29:33.854514 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.854507 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-tls-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:29:33.854514 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.854518 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2sbb\" (UniqueName: \"kubernetes.io/projected/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-kube-api-access-t2sbb\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:29:33.854735 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.854527 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-tmp-dir\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:29:33.854735 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.854536 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-model-cache\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:29:33.854735 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:33.854544 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/148cb8ac-d0e7-4fd8-9682-462f844cc7b2-home\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:29:34.279328 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:34.279297 2564 generic.go:356] "Generic (PLEG): container finished" podID="148cb8ac-d0e7-4fd8-9682-462f844cc7b2" containerID="ef35a4e0410071e009b96c31e13ba7eb8262672db5973b7319c2b0427953bbd7" exitCode=0 Jul 08 23:29:34.279465 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:34.279335 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" Jul 08 23:29:34.279465 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:34.279368 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" event={"ID":"148cb8ac-d0e7-4fd8-9682-462f844cc7b2","Type":"ContainerDied","Data":"ef35a4e0410071e009b96c31e13ba7eb8262672db5973b7319c2b0427953bbd7"} Jul 08 23:29:34.279465 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:34.279404 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m" event={"ID":"148cb8ac-d0e7-4fd8-9682-462f844cc7b2","Type":"ContainerDied","Data":"750c3592b9bb7aeecc9b6c9e94084ae3b293e63cc11ef45571a8c6ce1297027e"} Jul 08 23:29:34.279465 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:34.279420 2564 scope.go:117] "RemoveContainer" containerID="ef35a4e0410071e009b96c31e13ba7eb8262672db5973b7319c2b0427953bbd7" Jul 08 23:29:34.287716 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:34.287700 2564 scope.go:117] "RemoveContainer" containerID="ef35a4e0410071e009b96c31e13ba7eb8262672db5973b7319c2b0427953bbd7" Jul 08 23:29:34.287949 ip-10-0-143-210 kubenswrapper[2564]: E0708 23:29:34.287929 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef35a4e0410071e009b96c31e13ba7eb8262672db5973b7319c2b0427953bbd7\": container with ID starting with ef35a4e0410071e009b96c31e13ba7eb8262672db5973b7319c2b0427953bbd7 not found: ID does not exist" containerID="ef35a4e0410071e009b96c31e13ba7eb8262672db5973b7319c2b0427953bbd7" Jul 08 23:29:34.288000 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:34.287957 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef35a4e0410071e009b96c31e13ba7eb8262672db5973b7319c2b0427953bbd7"} err="failed to get container status \"ef35a4e0410071e009b96c31e13ba7eb8262672db5973b7319c2b0427953bbd7\": rpc error: code = NotFound desc = could not find container \"ef35a4e0410071e009b96c31e13ba7eb8262672db5973b7319c2b0427953bbd7\": container with ID starting with ef35a4e0410071e009b96c31e13ba7eb8262672db5973b7319c2b0427953bbd7 not found: ID does not exist" Jul 08 23:29:34.294898 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:34.294878 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m"] Jul 08 23:29:34.297979 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:34.297958 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-77dfb5/sub-test-model-77dfb5-kserve-79855df56f-q7q5m"] Jul 08 23:29:36.267539 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:36.267507 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="148cb8ac-d0e7-4fd8-9682-462f844cc7b2" path="/var/lib/kubelet/pods/148cb8ac-d0e7-4fd8-9682-462f844cc7b2/volumes" Jul 08 23:29:43.198656 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:43.198624 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:29:43.298199 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:43.298170 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:29:50.940762 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:50.940728 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:29:51.042824 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:51.042780 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:29:57.696210 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.696177 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm"] Jul 08 23:29:57.696775 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.696686 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e18472a-a777-49e3-b27b-6dc7a70ba516" containerName="main" Jul 08 23:29:57.696775 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.696702 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e18472a-a777-49e3-b27b-6dc7a70ba516" containerName="main" Jul 08 23:29:57.696775 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.696735 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="148cb8ac-d0e7-4fd8-9682-462f844cc7b2" containerName="main" Jul 08 23:29:57.696775 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.696744 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="148cb8ac-d0e7-4fd8-9682-462f844cc7b2" containerName="main" Jul 08 23:29:57.696974 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.696852 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="148cb8ac-d0e7-4fd8-9682-462f844cc7b2" containerName="main" Jul 08 23:29:57.696974 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.696866 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e18472a-a777-49e3-b27b-6dc7a70ba516" containerName="main" Jul 08 23:29:57.699957 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.699940 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.702409 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.702390 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5b1175\"/\"kube-root-ca.crt\"" Jul 08 23:29:57.703461 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.703443 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5b1175\"/\"openshift-service-ca.crt\"" Jul 08 23:29:57.703549 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.703495 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5b1175\"/\"default-dockercfg-4z8gf\"" Jul 08 23:29:57.703599 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.703554 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5b1175\"/\"rate-test-model-5b1175-kserve-self-signed-certs\"" Jul 08 23:29:57.709452 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.709431 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm"] Jul 08 23:29:57.732600 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.732566 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:29:57.757693 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.757671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-model-cache\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.757790 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.757710 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-tmp-dir\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.757790 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.757754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htm2w\" (UniqueName: \"kubernetes.io/projected/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-kube-api-access-htm2w\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.757867 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.757811 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-dshm\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.757867 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.757853 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-tls-certs\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.757936 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.757880 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-home\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.858638 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.858614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-tmp-dir\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.858739 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.858640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htm2w\" (UniqueName: \"kubernetes.io/projected/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-kube-api-access-htm2w\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.858739 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.858664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-dshm\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.858739 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.858710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-tls-certs\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.858853 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.858751 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-home\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.858908 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.858864 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-model-cache\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.859206 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.859186 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-tmp-dir\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.859309 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.859242 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-model-cache\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.859309 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.859289 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-home\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.860885 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.860865 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-dshm\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.861268 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.861243 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-tls-certs\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:57.866374 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:57.866354 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htm2w\" (UniqueName: \"kubernetes.io/projected/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-kube-api-access-htm2w\") pod \"rate-test-model-5b1175-kserve-6758794f95-m8llm\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:58.009797 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:58.009762 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:29:58.140269 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:58.140246 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm"] Jul 08 23:29:58.142212 ip-10-0-143-210 kubenswrapper[2564]: W0708 23:29:58.142186 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda23b830f_cee5_4b44_8e19_b5d1391e3f4b.slice/crio-2c4a445c483cd41bdc1dc841663579754cbd9c0b90bd0679600c1553a3489ede WatchSource:0}: Error finding container 2c4a445c483cd41bdc1dc841663579754cbd9c0b90bd0679600c1553a3489ede: Status 404 returned error can't find the container with id 2c4a445c483cd41bdc1dc841663579754cbd9c0b90bd0679600c1553a3489ede Jul 08 23:29:58.398988 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:58.398901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" event={"ID":"a23b830f-cee5-4b44-8e19-b5d1391e3f4b","Type":"ContainerStarted","Data":"6ae0aff5b9078022f624de07253b96672d258579b525f18ba828814c42bf3d9c"} Jul 08 23:29:58.398988 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:58.398943 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" event={"ID":"a23b830f-cee5-4b44-8e19-b5d1391e3f4b","Type":"ContainerStarted","Data":"2c4a445c483cd41bdc1dc841663579754cbd9c0b90bd0679600c1553a3489ede"} Jul 08 23:29:58.415722 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:29:58.415675 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" podStartSLOduration=1.415658205 podStartE2EDuration="1.415658205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:29:58.413824542 +0000 UTC m=+2592.669272249" watchObservedRunningTime="2026-07-08 23:29:58.415658205 +0000 UTC m=+2592.671105902" Jul 08 23:30:00.603591 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:00.603560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:30:02.989756 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:02.989725 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:30:08.010724 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:08.010630 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:30:08.010724 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:08.010690 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:30:08.023090 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:08.023059 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:30:08.462585 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:08.462511 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:30:24.162097 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:24.162044 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:30:27.479751 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:27.479712 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:30:31.203873 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:31.203841 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:30:47.207228 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:47.207189 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:30:47.275651 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:47.275618 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:30:52.981986 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:52.981953 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:30:53.893921 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:53.893890 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm"] Jul 08 23:30:53.912940 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:53.912914 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:30:54.814369 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:54.814338 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7578-140502206417472"] Jul 08 23:30:54.817755 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:54.817738 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140502206417472" Jul 08 23:30:54.819929 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:54.819907 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-njm98\"" Jul 08 23:30:54.826451 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:54.826428 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7578-140502206417472"] Jul 08 23:30:54.858205 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:54.858184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsm9t\" (UniqueName: \"kubernetes.io/projected/87e2db6f-bb70-4088-8430-9671cd97e5c2-kube-api-access-dsm9t\") pod \"test-curl-7578-140502206417472\" (UID: \"87e2db6f-bb70-4088-8430-9671cd97e5c2\") " pod="opendatahub/test-curl-7578-140502206417472" Jul 08 23:30:54.959425 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:54.959404 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsm9t\" (UniqueName: \"kubernetes.io/projected/87e2db6f-bb70-4088-8430-9671cd97e5c2-kube-api-access-dsm9t\") pod \"test-curl-7578-140502206417472\" (UID: \"87e2db6f-bb70-4088-8430-9671cd97e5c2\") " pod="opendatahub/test-curl-7578-140502206417472" Jul 08 23:30:54.967505 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:54.967485 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsm9t\" (UniqueName: \"kubernetes.io/projected/87e2db6f-bb70-4088-8430-9671cd97e5c2-kube-api-access-dsm9t\") pod \"test-curl-7578-140502206417472\" (UID: \"87e2db6f-bb70-4088-8430-9671cd97e5c2\") " pod="opendatahub/test-curl-7578-140502206417472" Jul 08 23:30:55.128286 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:55.128207 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140502206417472" Jul 08 23:30:55.256701 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:55.256676 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7578-140502206417472"] Jul 08 23:30:55.258881 ip-10-0-143-210 kubenswrapper[2564]: W0708 23:30:55.258854 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87e2db6f_bb70_4088_8430_9671cd97e5c2.slice/crio-1a71e6dcf5d5253ca86b15d55bb89b9d5caa27d3a417f03cea92f7d748621ca3 WatchSource:0}: Error finding container 1a71e6dcf5d5253ca86b15d55bb89b9d5caa27d3a417f03cea92f7d748621ca3: Status 404 returned error can't find the container with id 1a71e6dcf5d5253ca86b15d55bb89b9d5caa27d3a417f03cea92f7d748621ca3 Jul 08 23:30:55.260259 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:55.260241 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 23:30:55.693647 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:55.693614 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7578-140502206417472" event={"ID":"87e2db6f-bb70-4088-8430-9671cd97e5c2","Type":"ContainerStarted","Data":"1a71e6dcf5d5253ca86b15d55bb89b9d5caa27d3a417f03cea92f7d748621ca3"} Jul 08 23:30:56.700284 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:56.700252 2564 generic.go:356] "Generic (PLEG): container finished" podID="87e2db6f-bb70-4088-8430-9671cd97e5c2" containerID="a974df727d6388fed08f06099093f538581c3976465a20742d3969c3d5e92ae8" exitCode=6 Jul 08 23:30:56.700682 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:56.700329 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7578-140502206417472" event={"ID":"87e2db6f-bb70-4088-8430-9671cd97e5c2","Type":"ContainerDied","Data":"a974df727d6388fed08f06099093f538581c3976465a20742d3969c3d5e92ae8"} Jul 08 23:30:57.839490 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:57.839467 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140502206417472" Jul 08 23:30:57.858097 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:57.858058 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7578-140502206417472_87e2db6f-bb70-4088-8430-9671cd97e5c2/test-curl-7578-140502206417472/0.log" Jul 08 23:30:57.882738 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:57.882716 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7578-140502206417472"] Jul 08 23:30:57.885851 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:57.885833 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7578-140502206417472"] Jul 08 23:30:57.889162 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:57.889145 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsm9t\" (UniqueName: \"kubernetes.io/projected/87e2db6f-bb70-4088-8430-9671cd97e5c2-kube-api-access-dsm9t\") pod \"87e2db6f-bb70-4088-8430-9671cd97e5c2\" (UID: \"87e2db6f-bb70-4088-8430-9671cd97e5c2\") " Jul 08 23:30:57.891192 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:57.891170 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e2db6f-bb70-4088-8430-9671cd97e5c2-kube-api-access-dsm9t" (OuterVolumeSpecName: "kube-api-access-dsm9t") pod "87e2db6f-bb70-4088-8430-9671cd97e5c2" (UID: "87e2db6f-bb70-4088-8430-9671cd97e5c2"). InnerVolumeSpecName "kube-api-access-dsm9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 23:30:57.990376 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:57.990324 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsm9t\" (UniqueName: \"kubernetes.io/projected/87e2db6f-bb70-4088-8430-9671cd97e5c2-kube-api-access-dsm9t\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:30:58.273965 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:58.273934 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87e2db6f-bb70-4088-8430-9671cd97e5c2" path="/var/lib/kubelet/pods/87e2db6f-bb70-4088-8430-9671cd97e5c2/volumes" Jul 08 23:30:58.711418 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:58.711325 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140502206417472" Jul 08 23:30:58.711418 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:30:58.711379 2564 scope.go:117] "RemoveContainer" containerID="a974df727d6388fed08f06099093f538581c3976465a20742d3969c3d5e92ae8" Jul 08 23:31:06.480319 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:06.480277 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:31:08.914700 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:08.914654 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" podUID="a23b830f-cee5-4b44-8e19-b5d1391e3f4b" containerName="main" containerID="cri-o://6ae0aff5b9078022f624de07253b96672d258579b525f18ba828814c42bf3d9c" gracePeriod=45 Jul 08 23:31:09.177794 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.177739 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:31:09.289945 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.289914 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-home\") pod \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " Jul 08 23:31:09.290108 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.290016 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-tls-certs\") pod \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " Jul 08 23:31:09.290108 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.290058 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-tmp-dir\") pod \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " Jul 08 23:31:09.290209 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.290108 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-dshm\") pod \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " Jul 08 23:31:09.290209 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.290166 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htm2w\" (UniqueName: \"kubernetes.io/projected/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-kube-api-access-htm2w\") pod \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " Jul 08 23:31:09.290209 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.290194 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-model-cache\") pod \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\" (UID: \"a23b830f-cee5-4b44-8e19-b5d1391e3f4b\") " Jul 08 23:31:09.290331 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.290214 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-home" (OuterVolumeSpecName: "home") pod "a23b830f-cee5-4b44-8e19-b5d1391e3f4b" (UID: "a23b830f-cee5-4b44-8e19-b5d1391e3f4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:31:09.290372 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.290352 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a23b830f-cee5-4b44-8e19-b5d1391e3f4b" (UID: "a23b830f-cee5-4b44-8e19-b5d1391e3f4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:31:09.290516 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.290478 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a23b830f-cee5-4b44-8e19-b5d1391e3f4b" (UID: "a23b830f-cee5-4b44-8e19-b5d1391e3f4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:31:09.290593 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.290502 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-tmp-dir\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:31:09.290593 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.290547 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-home\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:31:09.292306 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.292282 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a23b830f-cee5-4b44-8e19-b5d1391e3f4b" (UID: "a23b830f-cee5-4b44-8e19-b5d1391e3f4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 23:31:09.292403 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.292332 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-dshm" (OuterVolumeSpecName: "dshm") pod "a23b830f-cee5-4b44-8e19-b5d1391e3f4b" (UID: "a23b830f-cee5-4b44-8e19-b5d1391e3f4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 23:31:09.292403 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.292368 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-kube-api-access-htm2w" (OuterVolumeSpecName: "kube-api-access-htm2w") pod "a23b830f-cee5-4b44-8e19-b5d1391e3f4b" (UID: "a23b830f-cee5-4b44-8e19-b5d1391e3f4b"). InnerVolumeSpecName "kube-api-access-htm2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 23:31:09.391067 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.391039 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-tls-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:31:09.391067 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.391063 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-dshm\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:31:09.391237 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.391073 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htm2w\" (UniqueName: \"kubernetes.io/projected/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-kube-api-access-htm2w\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:31:09.391237 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.391109 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b830f-cee5-4b44-8e19-b5d1391e3f4b-model-cache\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 08 23:31:09.779915 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.779877 2564 generic.go:356] "Generic (PLEG): container finished" podID="a23b830f-cee5-4b44-8e19-b5d1391e3f4b" containerID="6ae0aff5b9078022f624de07253b96672d258579b525f18ba828814c42bf3d9c" exitCode=0 Jul 08 23:31:09.780146 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.779936 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" event={"ID":"a23b830f-cee5-4b44-8e19-b5d1391e3f4b","Type":"ContainerDied","Data":"6ae0aff5b9078022f624de07253b96672d258579b525f18ba828814c42bf3d9c"} Jul 08 23:31:09.780146 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.779948 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" Jul 08 23:31:09.780146 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.779977 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm" event={"ID":"a23b830f-cee5-4b44-8e19-b5d1391e3f4b","Type":"ContainerDied","Data":"2c4a445c483cd41bdc1dc841663579754cbd9c0b90bd0679600c1553a3489ede"} Jul 08 23:31:09.780146 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.779997 2564 scope.go:117] "RemoveContainer" containerID="6ae0aff5b9078022f624de07253b96672d258579b525f18ba828814c42bf3d9c" Jul 08 23:31:09.789493 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.789474 2564 scope.go:117] "RemoveContainer" containerID="6ae0aff5b9078022f624de07253b96672d258579b525f18ba828814c42bf3d9c" Jul 08 23:31:09.789758 ip-10-0-143-210 kubenswrapper[2564]: E0708 23:31:09.789741 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae0aff5b9078022f624de07253b96672d258579b525f18ba828814c42bf3d9c\": container with ID starting with 6ae0aff5b9078022f624de07253b96672d258579b525f18ba828814c42bf3d9c not found: ID does not exist" containerID="6ae0aff5b9078022f624de07253b96672d258579b525f18ba828814c42bf3d9c" Jul 08 23:31:09.789800 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.789767 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae0aff5b9078022f624de07253b96672d258579b525f18ba828814c42bf3d9c"} err="failed to get container status \"6ae0aff5b9078022f624de07253b96672d258579b525f18ba828814c42bf3d9c\": rpc error: code = NotFound desc = could not find container \"6ae0aff5b9078022f624de07253b96672d258579b525f18ba828814c42bf3d9c\": container with ID starting with 6ae0aff5b9078022f624de07253b96672d258579b525f18ba828814c42bf3d9c not found: ID does not exist" Jul 08 23:31:09.801188 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.801165 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm"] Jul 08 23:31:09.804799 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:09.804781 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5b1175/rate-test-model-5b1175-kserve-6758794f95-m8llm"] Jul 08 23:31:10.267884 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:10.267853 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a23b830f-cee5-4b44-8e19-b5d1391e3f4b" path="/var/lib/kubelet/pods/a23b830f-cee5-4b44-8e19-b5d1391e3f4b/volumes" Jul 08 23:31:14.570608 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:14.570577 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:14.980005 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:14.979927 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:15.080430 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:15.080400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:15.237397 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:15.237315 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:20.681692 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:20.681661 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:31:20.870130 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:20.870097 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:20.970309 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:20.970226 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:21.126497 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:21.126463 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:21.494135 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:21.494105 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:22.000103 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:22.000056 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:22.184028 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:22.184002 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:22.284076 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:22.283977 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:22.546763 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:22.546683 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:27.858050 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:27.858019 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:27.958741 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:27.958710 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:28.123192 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:28.123102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:32.488452 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:32.488413 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jsfqw"] Jul 08 23:31:34.006432 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:34.006404 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86d776fd96-jtkxj_1e774b31-f7ae-404c-a1d0-93f89b2c00ab/manager/0.log" Jul 08 23:31:34.012514 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:34.012493 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86d776fd96-jtkxj_1e774b31-f7ae-404c-a1d0-93f89b2c00ab/copy-manifests/0.log" Jul 08 23:31:35.454032 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:35.454005 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-d2x6d_87168efd-e1a0-42e6-b466-10d867c4ed0e/manager/0.log" Jul 08 23:31:35.906796 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:35.906764 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jsfqw_9f518cf6-b12a-4943-b210-2bcf0cc459ed/limitador/0.log" Jul 08 23:31:36.695769 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:36.695740 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78df955549-w6fpf_361c10e9-fc5c-4f69-9647-ae45cebfe624/kube-auth-proxy/0.log" Jul 08 23:31:37.482989 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:37.482960 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-x9s2w_2bc58678-984d-4964-8300-5a26514f24a7/main/0.log" Jul 08 23:31:37.818208 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:37.818181 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db647vv_ff6984e4-3311-4b8f-8e5a-4421eefeba9e/main/0.log" Jul 08 23:31:38.039361 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:38.039288 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-g6cqh_7799ee64-f6f8-44ce-8147-eaa844c87254/main/0.log" Jul 08 23:31:42.269010 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:42.268981 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:42.368550 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:42.368502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" limit=1024 Jul 08 23:31:45.270252 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:45.270220 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-twk7h_0677a18f-b178-455f-a982-4c31ec52aab0/global-pull-secret-syncer/0.log" Jul 08 23:31:45.316061 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:45.316030 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-grqlf_e72ff1d4-bb03-44e5-80cb-8f8002a894e8/konnectivity-agent/0.log" Jul 08 23:31:45.431463 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:45.431438 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-210.ec2.internal_be9286b9c4dca1e8d98a40958054271d/haproxy/0.log" Jul 08 23:31:46.439789 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:46.439761 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:31:46.443530 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:46.443506 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 23:31:46.448634 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:46.448616 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:31:46.459991 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:46.459959 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pd2mh_63c5c2ac-8599-49be-b250-926ec607c77e/ovn-acl-logging/0.log" Jul 08 23:31:50.871678 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:50.871592 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-d2x6d_87168efd-e1a0-42e6-b466-10d867c4ed0e/manager/0.log" Jul 08 23:31:51.196844 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:51.196751 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jsfqw_9f518cf6-b12a-4943-b210-2bcf0cc459ed/limitador/0.log" Jul 08 23:31:53.042038 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.042002 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-9p9rm_7cb88df0-e9f8-4bec-9194-79356f674f21/cluster-monitoring-operator/0.log" Jul 08 23:31:53.065598 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.065572 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-zscp8_5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9/kube-state-metrics/0.log" Jul 08 23:31:53.084163 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.084137 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-zscp8_5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9/kube-rbac-proxy-main/0.log" Jul 08 23:31:53.103879 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.103858 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-zscp8_5ef646bd-7f85-4542-b4b9-b8c5cbcc5fd9/kube-rbac-proxy-self/0.log" Jul 08 23:31:53.129251 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.129225 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-587895c54b-tfndr_26e7d046-8455-4ac9-969f-c436e90f8520/metrics-server/0.log" Jul 08 23:31:53.181942 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.181886 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k56fx_2c84f58f-ff9a-4aa2-bbcf-5ea023174a99/node-exporter/0.log" Jul 08 23:31:53.209905 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.209890 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k56fx_2c84f58f-ff9a-4aa2-bbcf-5ea023174a99/kube-rbac-proxy/0.log" Jul 08 23:31:53.234134 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.234114 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k56fx_2c84f58f-ff9a-4aa2-bbcf-5ea023174a99/init-textfile/0.log" Jul 08 23:31:53.408709 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.408682 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-tc2ls_868a794a-7d4a-4c8c-82ed-e8cf93aecd4a/kube-rbac-proxy-main/0.log" Jul 08 23:31:53.435567 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.435498 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-tc2ls_868a794a-7d4a-4c8c-82ed-e8cf93aecd4a/kube-rbac-proxy-self/0.log" Jul 08 23:31:53.466940 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.466912 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-tc2ls_868a794a-7d4a-4c8c-82ed-e8cf93aecd4a/openshift-state-metrics/0.log" Jul 08 23:31:53.513720 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.513695 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a0b6fd7-dd32-4672-8c47-286e58651841/prometheus/0.log" Jul 08 23:31:53.533305 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.533279 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a0b6fd7-dd32-4672-8c47-286e58651841/config-reloader/0.log" Jul 08 23:31:53.556805 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.556783 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a0b6fd7-dd32-4672-8c47-286e58651841/thanos-sidecar/0.log" Jul 08 23:31:53.577045 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.577021 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a0b6fd7-dd32-4672-8c47-286e58651841/kube-rbac-proxy-web/0.log" Jul 08 23:31:53.600848 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.600827 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a0b6fd7-dd32-4672-8c47-286e58651841/kube-rbac-proxy/0.log" Jul 08 23:31:53.622584 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.622562 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a0b6fd7-dd32-4672-8c47-286e58651841/kube-rbac-proxy-thanos/0.log" Jul 08 23:31:53.642368 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.642347 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a0b6fd7-dd32-4672-8c47-286e58651841/init-config-reloader/0.log" Jul 08 23:31:53.718049 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.717974 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-j9692_4f41cd53-0ea2-49fa-8677-683e3abf8150/prometheus-operator-admission-webhook/0.log" Jul 08 23:31:53.854189 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.854158 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76d47b86df-n7r7k_ad17e873-f56b-4bf4-a664-2eff93d0aecb/thanos-query/0.log" Jul 08 23:31:53.873041 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.873015 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76d47b86df-n7r7k_ad17e873-f56b-4bf4-a664-2eff93d0aecb/kube-rbac-proxy-web/0.log" Jul 08 23:31:53.891407 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.891386 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76d47b86df-n7r7k_ad17e873-f56b-4bf4-a664-2eff93d0aecb/kube-rbac-proxy/0.log" Jul 08 23:31:53.913951 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.913929 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76d47b86df-n7r7k_ad17e873-f56b-4bf4-a664-2eff93d0aecb/prom-label-proxy/0.log" Jul 08 23:31:53.930450 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.930426 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76d47b86df-n7r7k_ad17e873-f56b-4bf4-a664-2eff93d0aecb/kube-rbac-proxy-rules/0.log" Jul 08 23:31:53.951208 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:53.951187 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76d47b86df-n7r7k_ad17e873-f56b-4bf4-a664-2eff93d0aecb/kube-rbac-proxy-metrics/0.log" Jul 08 23:31:55.182396 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.182367 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-pm98b_2d8cc6ed-a2e9-422d-ac36-2970b983252a/networking-console-plugin/0.log" Jul 08 23:31:55.769918 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.769888 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/1.log" Jul 08 23:31:55.774449 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.774428 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sblc2_7eebb5dd-85b2-45fa-8840-ab76361f826d/console-operator/2.log" Jul 08 23:31:55.883387 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.883355 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p"] Jul 08 23:31:55.883759 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.883746 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23b830f-cee5-4b44-8e19-b5d1391e3f4b" containerName="main" Jul 08 23:31:55.883806 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.883761 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23b830f-cee5-4b44-8e19-b5d1391e3f4b" containerName="main" Jul 08 23:31:55.883806 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.883792 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e2db6f-bb70-4088-8430-9671cd97e5c2" containerName="test-curl-7578-140502206417472" Jul 08 23:31:55.883806 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.883798 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e2db6f-bb70-4088-8430-9671cd97e5c2" containerName="test-curl-7578-140502206417472" Jul 08 23:31:55.883909 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.883862 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23b830f-cee5-4b44-8e19-b5d1391e3f4b" containerName="main" Jul 08 23:31:55.883909 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.883875 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e2db6f-bb70-4088-8430-9671cd97e5c2" containerName="test-curl-7578-140502206417472" Jul 08 23:31:55.887113 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.887073 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:55.889604 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.889583 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c9xpw\"/\"kube-root-ca.crt\"" Jul 08 23:31:55.890532 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.890511 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c9xpw\"/\"openshift-service-ca.crt\"" Jul 08 23:31:55.890532 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.890530 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c9xpw\"/\"default-dockercfg-nlb47\"" Jul 08 23:31:55.897384 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.897362 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p"] Jul 08 23:31:55.972337 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.972311 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gglxt\" (UniqueName: \"kubernetes.io/projected/a4b13f78-a260-4b15-bf17-5043994e0f02-kube-api-access-gglxt\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:55.972482 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.972350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b13f78-a260-4b15-bf17-5043994e0f02-lib-modules\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:55.972482 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.972379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4b13f78-a260-4b15-bf17-5043994e0f02-podres\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:55.972482 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.972447 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4b13f78-a260-4b15-bf17-5043994e0f02-proc\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:55.972588 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:55.972507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b13f78-a260-4b15-bf17-5043994e0f02-sys\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:56.073778 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:56.073677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4b13f78-a260-4b15-bf17-5043994e0f02-proc\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:56.073778 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:56.073758 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b13f78-a260-4b15-bf17-5043994e0f02-sys\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:56.074038 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:56.073798 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4b13f78-a260-4b15-bf17-5043994e0f02-proc\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:56.074038 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:56.073822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gglxt\" (UniqueName: \"kubernetes.io/projected/a4b13f78-a260-4b15-bf17-5043994e0f02-kube-api-access-gglxt\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:56.074038 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:56.073860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b13f78-a260-4b15-bf17-5043994e0f02-lib-modules\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:56.074038 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:56.073886 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b13f78-a260-4b15-bf17-5043994e0f02-sys\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:56.074038 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:56.073901 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4b13f78-a260-4b15-bf17-5043994e0f02-podres\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:56.074038 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:56.074004 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b13f78-a260-4b15-bf17-5043994e0f02-lib-modules\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:56.074038 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:56.074034 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4b13f78-a260-4b15-bf17-5043994e0f02-podres\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:56.084558 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:56.084536 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gglxt\" (UniqueName: \"kubernetes.io/projected/a4b13f78-a260-4b15-bf17-5043994e0f02-kube-api-access-gglxt\") pod \"perf-node-gather-daemonset-5ht8p\" (UID: \"a4b13f78-a260-4b15-bf17-5043994e0f02\") " pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:56.199097 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:56.199065 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:56.331885 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:56.331862 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p"] Jul 08 23:31:56.333918 ip-10-0-143-210 kubenswrapper[2564]: W0708 23:31:56.333894 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda4b13f78_a260_4b15_bf17_5043994e0f02.slice/crio-4d381c747a7bd73dcf0fa5afdef79b3489c680dbaabf567495e1a15c1637f46d WatchSource:0}: Error finding container 4d381c747a7bd73dcf0fa5afdef79b3489c680dbaabf567495e1a15c1637f46d: Status 404 returned error can't find the container with id 4d381c747a7bd73dcf0fa5afdef79b3489c680dbaabf567495e1a15c1637f46d Jul 08 23:31:56.451843 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:56.451822 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-fdz8x_1ef9ab8d-16ec-4e0e-a553-87cee0316d49/download-server/0.log" Jul 08 23:31:57.011005 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:57.010969 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-xtx76_74c83256-c7b1-4ad4-8ba9-f8b734a2ff4d/volume-data-source-validator/0.log" Jul 08 23:31:57.011210 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:57.011180 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" event={"ID":"a4b13f78-a260-4b15-bf17-5043994e0f02","Type":"ContainerStarted","Data":"421ac2c56d0b7b205255bd30e1b4af4ed2656104c725b2e50ef3563f1a3bb41d"} Jul 08 23:31:57.011270 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:57.011226 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" event={"ID":"a4b13f78-a260-4b15-bf17-5043994e0f02","Type":"ContainerStarted","Data":"4d381c747a7bd73dcf0fa5afdef79b3489c680dbaabf567495e1a15c1637f46d"} Jul 08 23:31:57.011270 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:57.011251 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:31:57.026608 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:57.026564 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" podStartSLOduration=2.026551434 podStartE2EDuration="2.026551434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 23:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:31:57.024388247 +0000 UTC m=+2711.279835951" watchObservedRunningTime="2026-07-08 23:31:57.026551434 +0000 UTC m=+2711.281999137" Jul 08 23:31:58.076294 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:58.076248 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xv6dw_407a5c0c-24fd-474d-b677-65c32d5c1ca5/dns/0.log" Jul 08 23:31:58.094950 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:58.094926 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xv6dw_407a5c0c-24fd-474d-b677-65c32d5c1ca5/kube-rbac-proxy/0.log" Jul 08 23:31:58.116052 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:58.116019 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-297fk_eaf82025-12e7-4d8d-a76a-127dd040d050/dns-node-resolver/0.log" Jul 08 23:31:58.746177 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:58.746150 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h7dmt_9c0a262a-d0f9-4fd3-91dc-44848d6d50b2/node-ca/0.log" Jul 08 23:31:59.918734 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:31:59.918706 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78df955549-w6fpf_361c10e9-fc5c-4f69-9647-ae45cebfe624/kube-auth-proxy/0.log" Jul 08 23:32:00.801030 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:00.800989 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jpnsr_32679a71-0c9d-4b6a-8b60-130bd59ae45c/serve-healthcheck-canary/0.log" Jul 08 23:32:01.317942 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:01.317884 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-77fdr_e1cc7e87-c580-4f51-8870-9ed1cbe58166/kube-rbac-proxy/0.log" Jul 08 23:32:01.340510 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:01.340482 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-77fdr_e1cc7e87-c580-4f51-8870-9ed1cbe58166/exporter/0.log" Jul 08 23:32:01.361249 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:01.361224 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-77fdr_e1cc7e87-c580-4f51-8870-9ed1cbe58166/extractor/0.log" Jul 08 23:32:03.026916 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:03.026890 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c9xpw/perf-node-gather-daemonset-5ht8p" Jul 08 23:32:04.194379 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:04.194351 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86d776fd96-jtkxj_1e774b31-f7ae-404c-a1d0-93f89b2c00ab/manager/0.log" Jul 08 23:32:04.212854 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:04.212830 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86d776fd96-jtkxj_1e774b31-f7ae-404c-a1d0-93f89b2c00ab/copy-manifests/0.log" Jul 08 23:32:05.700128 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:05.700023 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cfd684dd8-dllmj_e5a0819b-b6e8-40cf-ab85-76c8991b268e/manager/0.log" Jul 08 23:32:05.754134 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:05.754114 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-5642z_9615fd53-e65a-48ac-a76e-d769f2194c6e/openshift-lws-operator/0.log" Jul 08 23:32:06.529157 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:06.529121 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78df955549-w6fpf_361c10e9-fc5c-4f69-9647-ae45cebfe624/kube-auth-proxy/0.log" Jul 08 23:32:08.172991 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:08.172962 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78df955549-w6fpf_361c10e9-fc5c-4f69-9647-ae45cebfe624/kube-auth-proxy/0.log" Jul 08 23:32:10.842166 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:10.842142 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-67rtd_35a16af7-2907-4ca1-a490-c0450ef37cbe/migrator/0.log" Jul 08 23:32:10.860787 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:10.860760 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-67rtd_35a16af7-2907-4ca1-a490-c0450ef37cbe/graceful-termination/0.log" Jul 08 23:32:11.198562 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:11.198466 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-xgdbp_1e961afb-3fe2-4fad-9c78-45b9370b073d/kube-storage-version-migrator-operator/1.log" Jul 08 23:32:11.199326 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:11.199307 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-xgdbp_1e961afb-3fe2-4fad-9c78-45b9370b073d/kube-storage-version-migrator-operator/0.log" Jul 08 23:32:12.368404 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:12.368374 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gjw2b_3168d934-7d87-498f-9733-8059ff036149/kube-multus-additional-cni-plugins/0.log" Jul 08 23:32:12.387444 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:12.387421 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gjw2b_3168d934-7d87-498f-9733-8059ff036149/egress-router-binary-copy/0.log" Jul 08 23:32:12.406412 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:12.406396 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gjw2b_3168d934-7d87-498f-9733-8059ff036149/cni-plugins/0.log" Jul 08 23:32:12.425278 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:12.425260 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gjw2b_3168d934-7d87-498f-9733-8059ff036149/bond-cni-plugin/0.log" Jul 08 23:32:12.445769 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:12.445751 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gjw2b_3168d934-7d87-498f-9733-8059ff036149/routeoverride-cni/0.log" Jul 08 23:32:12.465212 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:12.465195 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gjw2b_3168d934-7d87-498f-9733-8059ff036149/whereabouts-cni-bincopy/0.log" Jul 08 23:32:12.487823 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:12.487800 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gjw2b_3168d934-7d87-498f-9733-8059ff036149/whereabouts-cni/0.log" Jul 08 23:32:12.651364 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:12.651285 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-j6kcc_5c226b53-16d4-436d-8cae-e37ab29dafeb/kube-multus/0.log" Jul 08 23:32:12.751159 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:12.751128 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rldqz_4ed6eae7-7b43-47ba-b489-136fab2a8ade/network-metrics-daemon/0.log" Jul 08 23:32:12.768674 ip-10-0-143-210 kubenswrapper[2564]: I0708 23:32:12.768647 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rldqz_4ed6eae7-7b43-47ba-b489-136fab2a8ade/kube-rbac-proxy/0.log"