Jul 14 19:10:49.502639 ip-10-0-138-70 systemd[1]: Starting Kubernetes Kubelet... Jul 14 19:10:49.896772 ip-10-0-138-70 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:10:49.896772 ip-10-0-138-70 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 19:10:49.896772 ip-10-0-138-70 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:10:49.896772 ip-10-0-138-70 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 19:10:49.896772 ip-10-0-138-70 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:10:49.897470 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.896843 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 19:10:49.902180 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902001 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:10:49.902180 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902182 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902186 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902189 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902192 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902196 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902199 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902202 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902204 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902207 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902209 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902212 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902215 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902217 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902220 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902222 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902225 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902228 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902233 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902235 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:10:49.902261 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902238 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902241 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902244 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902247 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902249 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902252 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902254 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902257 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902260 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902262 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902265 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902268 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902271 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902276 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902281 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902284 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902287 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902290 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902293 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:10:49.902729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902297 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902299 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902302 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902305 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902307 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902310 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902312 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902330 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902332 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902335 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902338 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902341 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902344 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902346 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902349 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902352 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902355 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902358 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902361 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902363 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:10:49.903192 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902366 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902369 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902372 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902374 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902377 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902379 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902383 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902385 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902388 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902390 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902393 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902396 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902399 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902402 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902404 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902407 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902409 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902412 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902414 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902417 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:10:49.903681 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902419 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902422 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902427 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902431 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902434 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902438 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902443 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902446 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902449 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902452 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902454 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902457 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902459 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902462 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902464 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902467 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902470 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902473 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902475 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:10:49.904159 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.902478 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903223 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903229 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903232 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903237 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903241 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903245 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903248 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903250 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903253 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903256 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903258 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903261 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903263 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903266 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903268 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903271 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903274 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903276 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903280 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:10:49.904642 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903283 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903285 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903288 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903291 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903294 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903296 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903299 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903301 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903303 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903306 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903308 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903311 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903313 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903332 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903335 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903337 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903340 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903343 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903345 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:10:49.905137 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903349 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903352 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903355 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903358 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903360 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903363 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903366 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903368 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903371 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903373 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903376 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903379 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903382 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903385 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903387 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903389 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903394 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903397 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903402 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:10:49.905615 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903405 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903408 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903411 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903413 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903416 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903419 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903421 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903425 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903427 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903429 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903433 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903437 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903439 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903443 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903446 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903449 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903452 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903455 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903458 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:10:49.906074 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903460 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903463 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903465 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903468 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903471 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903474 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903478 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903480 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903483 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903485 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903488 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903491 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903494 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903497 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903500 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903503 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903505 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903508 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903511 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903514 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:10:49.906568 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903516 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903519 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903521 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903954 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903960 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903964 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903967 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903970 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903973 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903976 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903978 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903981 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903985 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903987 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903990 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903993 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903996 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.903999 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904003 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904005 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:10:49.907065 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904008 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904011 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904014 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904017 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904020 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904022 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904025 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904027 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904030 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904033 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904035 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904038 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904041 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904044 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904047 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904049 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904052 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904055 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904058 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904061 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:10:49.907577 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904064 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904066 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904069 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904072 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904074 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904076 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904079 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904082 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904084 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904087 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904090 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904095 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904099 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904102 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904105 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904108 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904110 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904113 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904116 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:10:49.908053 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904118 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904121 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904123 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904126 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904128 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904131 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904133 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904136 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904138 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904141 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904143 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904146 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904148 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904151 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904154 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904156 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904159 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904162 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904164 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904167 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:10:49.908527 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904170 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904173 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904179 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904183 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904188 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904191 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904193 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904196 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904198 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904201 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904203 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904206 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904209 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904213 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904216 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904219 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904222 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904225 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904227 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:10:49.909050 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904230 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904233 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904236 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.904238 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904327 2572 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904344 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904351 2572 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904355 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904360 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904364 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904368 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904373 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904377 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904380 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904383 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904389 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904393 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904396 2572 flags.go:64] FLAG: --cgroup-root="" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904399 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904403 2572 flags.go:64] FLAG: --client-ca-file="" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904405 2572 flags.go:64] FLAG: --cloud-provider="external" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904409 2572 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904413 2572 flags.go:64] FLAG: --cluster-domain="" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904416 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 19:10:49.909535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904419 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904422 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904426 2572 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904430 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904433 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904436 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904439 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904443 2572 flags.go:64] FLAG: --contention-profiling="false" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904446 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904449 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904452 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904455 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904459 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904462 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904465 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904468 2572 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904471 2572 flags.go:64] FLAG: --enable-server="true" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904475 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904481 2572 flags.go:64] FLAG: --event-burst="100" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904485 2572 flags.go:64] FLAG: --event-qps="50" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904488 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904491 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904494 2572 flags.go:64] FLAG: --eviction-hard="" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904501 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 19:10:49.910206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904504 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904507 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904510 2572 flags.go:64] FLAG: --eviction-soft="" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904514 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904517 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904520 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904523 2572 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904526 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904529 2572 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904532 2572 flags.go:64] FLAG: --feature-gates="" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904536 2572 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904539 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904543 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904546 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904549 2572 flags.go:64] FLAG: --healthz-port="10248" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904552 2572 flags.go:64] FLAG: --help="false" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904555 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-138-70.ec2.internal" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904558 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904561 2572 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904564 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904567 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904571 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904574 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 19:10:49.910858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904577 2572 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904588 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904592 2572 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904598 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904602 2572 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904605 2572 flags.go:64] FLAG: --kube-reserved="" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904608 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904611 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904615 2572 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904618 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904621 2572 flags.go:64] FLAG: --lock-file="" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904624 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904627 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904630 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904635 2572 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904638 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904642 2572 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904645 2572 flags.go:64] FLAG: --logging-format="text" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904648 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904651 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904654 2572 flags.go:64] FLAG: --manifest-url="" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904658 2572 flags.go:64] FLAG: --manifest-url-header="" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904662 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904665 2572 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904670 2572 flags.go:64] FLAG: --max-pods="110" Jul 14 19:10:49.911473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904673 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904676 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904679 2572 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904682 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904685 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904688 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904691 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904700 2572 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904703 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904706 2572 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904711 2572 flags.go:64] FLAG: --pod-cidr="" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904715 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904722 2572 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904725 2572 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904729 2572 flags.go:64] FLAG: --pods-per-core="0" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904732 2572 flags.go:64] FLAG: --port="10250" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904736 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904739 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05165ae89d41f7df3" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904742 2572 flags.go:64] FLAG: --qos-reserved="" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904745 2572 flags.go:64] FLAG: --read-only-port="10255" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904749 2572 flags.go:64] FLAG: --register-node="true" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904752 2572 flags.go:64] FLAG: --register-with-taints="" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904760 2572 flags.go:64] FLAG: --registry-burst="10" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904763 2572 flags.go:64] FLAG: --registry-qps="5" Jul 14 19:10:49.912082 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904766 2572 flags.go:64] FLAG: --reserved-cpus="" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904770 2572 flags.go:64] FLAG: --reserved-memory="" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904775 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904779 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904782 2572 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904785 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904788 2572 flags.go:64] FLAG: --runonce="false" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904791 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904794 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904797 2572 flags.go:64] FLAG: --seccomp-default="false" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904800 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904803 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904807 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904810 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904813 2572 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904816 2572 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904819 2572 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904821 2572 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904826 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904829 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904833 2572 flags.go:64] FLAG: --system-cgroups="" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904836 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904841 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904844 2572 flags.go:64] FLAG: --tls-cert-file="" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904848 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 19:10:49.912686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904854 2572 flags.go:64] FLAG: --tls-min-version="" Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904857 2572 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904859 2572 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904863 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904865 2572 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904869 2572 flags.go:64] FLAG: --v="2" Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904873 2572 flags.go:64] FLAG: --version="false" Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904877 2572 flags.go:64] FLAG: --vmodule="" Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904881 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.904884 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905004 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905008 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905011 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905014 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905017 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905020 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905023 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905026 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905029 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905031 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905034 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905037 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:10:49.913313 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905039 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905042 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905045 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905049 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905052 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905055 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905058 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905061 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905064 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905067 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905070 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905073 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905076 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905078 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905081 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905084 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905086 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905089 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905092 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:10:49.913866 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905094 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905097 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905100 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905102 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905105 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905107 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905110 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905113 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905115 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905119 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905123 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905126 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905129 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905132 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905134 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905137 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905141 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905144 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905147 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:10:49.914459 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905150 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905153 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905156 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905161 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905164 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905167 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905169 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905172 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905175 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905177 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905180 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905182 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905185 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905188 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905190 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905193 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905196 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905198 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905201 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905203 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:10:49.914922 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905206 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905208 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905211 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905215 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905218 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905221 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905224 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905227 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905230 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905234 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905237 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905240 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905242 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905245 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905248 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905251 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905254 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905257 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905259 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:10:49.915415 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905262 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:10:49.915877 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905265 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:10:49.915877 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905269 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:10:49.915877 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905272 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:10:49.915877 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905274 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:10:49.915877 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905277 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:10:49.915877 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905280 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:10:49.915877 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905283 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:10:49.915877 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905285 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:10:49.915877 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.905288 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:10:49.915877 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.906141 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:10:49.915877 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.912787 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 19:10:49.915877 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.912806 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 19:10:49.915877 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.913671 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 19:10:49.915877 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.913678 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913743 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913748 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913753 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913757 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913760 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913763 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913767 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913770 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913775 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913777 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913780 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913783 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913786 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913789 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913792 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913795 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913798 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913800 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:10:49.916233 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913803 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913805 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913809 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913812 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913814 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913817 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913820 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913822 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913825 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913827 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913830 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913833 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913836 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913839 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913842 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913845 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913847 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913849 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913852 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913855 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:10:49.916729 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913857 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913860 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913862 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913865 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913868 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913871 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913874 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913878 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913880 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913883 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913886 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913888 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913891 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913893 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913896 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913898 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913901 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913904 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913908 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:10:49.917210 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913911 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913914 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913917 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913920 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913924 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913927 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913930 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913932 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913935 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913938 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913940 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913943 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913945 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913948 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913951 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913953 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913956 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913958 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913961 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913963 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:10:49.917725 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913966 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913969 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913972 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913974 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913977 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913979 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913982 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913984 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913986 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913989 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913991 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913994 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913996 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.913999 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914001 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914004 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914007 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914010 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914013 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914016 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:10:49.918202 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914020 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914024 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.914029 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914151 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914156 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914159 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914162 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914165 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914168 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914171 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914174 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914177 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914180 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914183 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914186 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:10:49.918745 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914189 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914192 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914195 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914197 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914200 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914203 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914205 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914208 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914211 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914215 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914219 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914221 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914224 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914227 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914231 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914233 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914236 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914239 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914242 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:10:49.919151 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914244 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914247 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914249 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914252 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914254 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914257 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914260 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914262 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914264 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914267 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914269 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914272 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914274 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914278 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914281 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914284 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914286 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914289 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914292 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:10:49.919698 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914295 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914298 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914301 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914303 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914306 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914309 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914313 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914342 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914347 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914352 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914355 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914358 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914361 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914364 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914367 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914370 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914372 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914375 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914378 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914381 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:10:49.920168 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914383 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914386 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914388 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914391 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914394 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914396 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914399 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914402 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914405 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914408 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914410 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914413 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914415 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914418 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914420 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914423 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914425 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914427 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914430 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914433 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:10:49.920709 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914436 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:10:49.921279 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914439 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:10:49.921279 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914441 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:10:49.921279 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914444 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:10:49.921279 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914446 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:10:49.921279 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914449 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:10:49.921279 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914451 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:10:49.921279 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914454 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:10:49.921279 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:49.914456 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:10:49.921279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.914462 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:10:49.921279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.914602 2572 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 19:10:49.921279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.917382 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 19:10:49.921279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.918608 2572 server.go:1019] "Starting client certificate rotation" Jul 14 19:10:49.921279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.918735 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 19:10:49.921279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.919569 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 19:10:49.921876 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.921028 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 19:10:49.922839 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.922346 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 19:10:49.937406 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.937379 2572 log.go:25] "Validated CRI v1 runtime API" Jul 14 19:10:49.939454 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.939436 2572 log.go:25] "Validated CRI v1 image API" Jul 14 19:10:49.940182 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.940164 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 19:10:49.942555 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.942529 2572 fs.go:135] Filesystem UUIDs: map[3a971127-54bd-4ecd-8031-09c25c24f6f4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e165bc83-c104-415f-baad-8e5b24fbfa69:/dev/nvme0n1p3] Jul 14 19:10:49.942644 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.942553 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 19:10:49.946548 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.946525 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 19:10:49.947099 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.946980 2572 manager.go:216] Machine: {Timestamp:2026-07-14 19:10:49.946424826 +0000 UTC m=+0.330218770 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100080 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21a3cd0d4e64139546f3be2b8b4e50 SystemUUID:ec21a3cd-0d4e-6413-9546-f3be2b8b4e50 BootID:96bfdac7-cad7-47d5-921b-fd4ea959e56e Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:be:6a:04:95:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:be:6a:04:95:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:8c:53:4e:5f:f5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 19:10:49.947099 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.947094 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 19:10:49.947303 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.947285 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 19:10:49.950038 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.950003 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 19:10:49.950190 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.950039 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-70.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 19:10:49.950805 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.950792 2572 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 19:10:49.950838 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.950807 2572 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 19:10:49.950838 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.950816 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 19:10:49.951595 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.951583 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 19:10:49.952357 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.952345 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 19:10:49.955138 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.955124 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 14 19:10:49.955263 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.955253 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 19:10:49.955305 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.955286 2572 kubelet.go:486] "Attempting to sync node with API server" Jul 14 19:10:49.955305 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.955297 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 19:10:49.955395 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.955309 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 19:10:49.955395 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.955335 2572 kubelet.go:398] "Adding apiserver pod source" Jul 14 19:10:49.955395 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.955344 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 19:10:49.955741 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.955729 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 19:10:49.955770 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.955749 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 19:10:49.958389 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.958371 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 19:10:49.960182 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.960167 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 19:10:49.960244 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.960200 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 19:10:49.961243 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.961230 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 19:10:49.961279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.961250 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 19:10:49.961279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.961257 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 19:10:49.961279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.961262 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 19:10:49.961279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.961273 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 19:10:49.961279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.961278 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 19:10:49.961427 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.961286 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 19:10:49.961427 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.961291 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 19:10:49.961427 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.961299 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 19:10:49.961427 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.961304 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 19:10:49.961427 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.961312 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 19:10:49.961427 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.961331 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 19:10:49.962073 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.962062 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 19:10:49.962073 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.962072 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 19:10:49.964573 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.964553 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-70.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 19:10:49.964665 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:49.964630 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 19:10:49.964731 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:49.964695 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-70.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 19:10:49.965714 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.965699 2572 server.go:1268] "Started kubelet" Jul 14 19:10:49.965778 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.965752 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 19:10:49.970669 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.970635 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 19:10:49.970759 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.970699 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 19:10:49.971220 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.971184 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mmvb8" Jul 14 19:10:49.971518 ip-10-0-138-70 systemd[1]: Started Kubernetes Kubelet. Jul 14 19:10:49.973159 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.973138 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 19:10:49.974026 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.973993 2572 server.go:310] "Adding debug handlers to kubelet server" Jul 14 19:10:49.975152 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:49.974247 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-70.ec2.internal.18c23db4fe84b272 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-70.ec2.internal,UID:ip-10-0-138-70.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-70.ec2.internal,},FirstTimestamp:2026-07-14 19:10:49.965654642 +0000 UTC m=+0.349448638,LastTimestamp:2026-07-14 19:10:49.965654642 +0000 UTC m=+0.349448638,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-70.ec2.internal,}" Jul 14 19:10:49.976572 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:49.976555 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 19:10:49.976647 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.976635 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mmvb8" Jul 14 19:10:49.980393 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.980378 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 19:10:49.980474 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.980393 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 19:10:49.980474 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:49.980451 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:49.981094 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.981076 2572 factory.go:55] Registering systemd factory Jul 14 19:10:49.981094 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.981096 2572 factory.go:223] Registration of the systemd container factory successfully Jul 14 19:10:49.981199 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.981130 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 19:10:49.981199 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.981144 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 19:10:49.981199 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.981157 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 19:10:49.981343 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.981250 2572 reconstruct.go:97] "Volume reconstruction finished" Jul 14 19:10:49.981343 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.981261 2572 reconciler.go:29] "Reconciler: start to sync state" Jul 14 19:10:49.981443 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.981346 2572 factory.go:153] Registering CRI-O factory Jul 14 19:10:49.981443 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.981358 2572 factory.go:223] Registration of the crio container factory successfully Jul 14 19:10:49.981443 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.981409 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 19:10:49.981443 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.981433 2572 factory.go:103] Registering Raw factory Jul 14 19:10:49.981586 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.981448 2572 manager.go:1196] Started watching for new ooms in manager Jul 14 19:10:49.981852 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.981839 2572 manager.go:349] Starting recovery of all containers Jul 14 19:10:49.991120 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.990889 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:10:49.992042 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.992022 2572 manager.go:354] Recovery completed Jul 14 19:10:49.996403 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:49.996386 2572 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-70.ec2.internal\" not found" node="ip-10-0-138-70.ec2.internal" Jul 14 19:10:49.996472 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.996444 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:49.999356 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.999339 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:49.999420 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.999371 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:49.999420 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.999384 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:49.999878 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.999865 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 19:10:49.999931 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.999878 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 19:10:49.999931 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.999907 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 14 19:10:49.999994 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:49.999928 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:50.001059 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.001036 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:50.001147 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.001077 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:50.001147 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.001093 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:50.002863 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.002847 2572 policy_none.go:49] "None policy: Start" Jul 14 19:10:50.002863 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.002866 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 19:10:50.002955 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.002878 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 19:10:50.004012 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.004002 2572 policy_none.go:47] "Start" Jul 14 19:10:50.042868 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.042852 2572 manager.go:337] "Starting Device Plugin manager" Jul 14 19:10:50.042952 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.042894 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 19:10:50.042952 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.042904 2572 server.go:87] "Starting device plugin registration server" Jul 14 19:10:50.043193 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.043180 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 19:10:50.043259 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.043195 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 19:10:50.043337 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.043290 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 19:10:50.043388 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.043380 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 19:10:50.043431 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.043388 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 19:10:50.043898 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.043865 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 19:10:50.043988 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.043920 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:50.143855 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.143817 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:50.144901 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.144886 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:50.144955 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.144918 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:50.144955 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.144929 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:50.145025 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.144955 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.153202 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.153149 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.153202 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.153175 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-70.ec2.internal\": node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:50.162257 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.162230 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 19:10:50.163488 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.163466 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 19:10:50.163594 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.163513 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 19:10:50.163594 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.163543 2572 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 19:10:50.163707 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.163653 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 19:10:50.167187 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.167165 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:50.167438 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.167424 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:10:50.264822 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.264783 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-70.ec2.internal"] Jul 14 19:10:50.264952 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.264871 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:50.265867 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.265848 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:50.265935 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.265890 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:50.265935 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.265905 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:50.267075 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.267050 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:50.267181 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.267160 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.267237 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.267188 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:50.267237 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.267209 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:50.267855 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.267839 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:50.267928 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.267871 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:50.267928 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.267842 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:50.267928 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.267906 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:50.267928 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.267919 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:50.267928 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.267884 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:50.269124 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.269108 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.269193 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.269136 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:50.269823 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.269810 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:50.269884 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.269833 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:50.269884 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.269843 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:50.291366 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.291338 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-70.ec2.internal\" not found" node="ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.296029 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.296010 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-70.ec2.internal\" not found" node="ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.367915 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.367880 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:50.383219 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.383191 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71fcd049ac89594ddba0d96bd90cd93b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal\" (UID: \"71fcd049ac89594ddba0d96bd90cd93b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.383312 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.383222 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71fcd049ac89594ddba0d96bd90cd93b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal\" (UID: \"71fcd049ac89594ddba0d96bd90cd93b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.383312 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.383243 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b04179ee16bc51140974d528cab249f3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-70.ec2.internal\" (UID: \"b04179ee16bc51140974d528cab249f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.468703 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.468623 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:50.484074 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.484040 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71fcd049ac89594ddba0d96bd90cd93b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal\" (UID: \"71fcd049ac89594ddba0d96bd90cd93b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.484175 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.484082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71fcd049ac89594ddba0d96bd90cd93b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal\" (UID: \"71fcd049ac89594ddba0d96bd90cd93b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.484175 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.484106 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b04179ee16bc51140974d528cab249f3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-70.ec2.internal\" (UID: \"b04179ee16bc51140974d528cab249f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.484175 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.484153 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b04179ee16bc51140974d528cab249f3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-70.ec2.internal\" (UID: \"b04179ee16bc51140974d528cab249f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.484175 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.484156 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71fcd049ac89594ddba0d96bd90cd93b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal\" (UID: \"71fcd049ac89594ddba0d96bd90cd93b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.484312 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.484159 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71fcd049ac89594ddba0d96bd90cd93b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal\" (UID: \"71fcd049ac89594ddba0d96bd90cd93b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.569234 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.569194 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:50.593682 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.593659 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.598551 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.598523 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.ec2.internal" Jul 14 19:10:50.670186 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.670143 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:50.770708 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.770638 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:50.871222 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.871181 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:50.918721 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.918690 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 19:10:50.919356 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.918863 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 19:10:50.919356 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.918879 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 19:10:50.971512 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:50.971482 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:50.981764 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.981733 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 19:05:50 +0000 UTC" deadline="2028-02-18 20:05:32.801306876 +0000 UTC" Jul 14 19:10:50.981764 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:50.981759 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14016h54m41.819551186s" Jul 14 19:10:51.001717 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:51.001693 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:10:51.071964 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:51.071883 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:51.172310 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:51.172282 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:51.272824 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:51.272786 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:51.373410 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:51.373347 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-70.ec2.internal\" not found" Jul 14 19:10:51.409233 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:51.409204 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:10:51.480912 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:51.480888 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal" Jul 14 19:10:51.492614 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:51.492578 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 19:10:51.493688 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:51.493672 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.ec2.internal" Jul 14 19:10:51.502053 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:51.502036 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 19:10:52.975571 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.975376 2572 apiserver.go:52] "Watching apiserver" Jul 14 19:10:52.984440 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.983907 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 19:10:52.986232 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.986206 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-dwftk","openshift-multus/network-metrics-daemon-h7v99","openshift-network-diagnostics/network-check-target-h5x8l","openshift-network-operator/iptables-alerter-l65x7","openshift-ovn-kubernetes/ovnkube-node-xw4dp","kube-system/global-pull-secret-syncer-t6nzh","kube-system/konnectivity-agent-88fh2","kube-system/kube-apiserver-proxy-ip-10-0-138-70.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v","openshift-dns/node-resolver-vq5gr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal","openshift-multus/multus-additional-cni-plugins-tbjqz","openshift-cluster-node-tuning-operator/tuned-p7l7s","openshift-image-registry/node-ca-gndd8"] Jul 14 19:10:52.987494 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.987477 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-88fh2" Jul 14 19:10:52.988510 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.988490 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:52.989456 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.989438 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:10:52.989525 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:52.989505 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:10:52.989827 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.989815 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 19:10:52.990024 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.990010 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 19:10:52.990147 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.990131 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-svlgl\"" Jul 14 19:10:52.990503 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.990490 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l65x7" Jul 14 19:10:52.990881 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.990864 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 19:10:52.991207 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.991192 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:10:52.991249 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.991213 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mh4bp\"" Jul 14 19:10:52.991664 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.991648 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:52.992747 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.992698 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:52.992840 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.992751 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:10:52.992840 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:52.992762 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:10:52.992840 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.992817 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 19:10:52.993007 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.992821 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2vrb6\"" Jul 14 19:10:52.993107 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.993093 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 19:10:52.993690 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.993677 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:52.994134 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.994117 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 19:10:52.994340 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.994326 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 19:10:52.994435 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.994421 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 19:10:52.994724 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.994711 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dwftk" Jul 14 19:10:52.995070 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.995034 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 19:10:52.995154 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.995061 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 19:10:52.995154 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.995072 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 19:10:52.995154 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.995127 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4zgtr\"" Jul 14 19:10:52.995833 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.995814 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:10:52.995909 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.995892 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 19:10:52.995909 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:52.995888 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:10:52.996261 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.996246 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 19:10:52.996436 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.996425 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 19:10:52.996600 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.996588 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fgkbc\"" Jul 14 19:10:52.996876 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.996864 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 19:10:52.996920 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.996883 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 19:10:52.996987 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.996971 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 19:10:52.997032 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.997019 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vq5gr" Jul 14 19:10:52.997107 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.997068 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-45f4p\"" Jul 14 19:10:52.997369 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.997354 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 19:10:52.998292 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.998278 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gndd8" Jul 14 19:10:52.999244 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.999229 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 19:10:52.999408 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.999244 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j2b4k\"" Jul 14 19:10:52.999408 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.999230 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 19:10:52.999537 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.999437 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 19:10:52.999537 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:52.999493 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.000224 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.000196 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 19:10:53.000296 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.000250 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 19:10:53.000461 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.000445 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 19:10:53.000655 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.000642 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 19:10:53.000791 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.000779 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l9hsf\"" Jul 14 19:10:53.001901 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.001884 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 19:10:53.002004 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.001906 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 19:10:53.002004 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.001947 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f2wdd\"" Jul 14 19:10:53.010325 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.010308 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-tuned\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.010403 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.010346 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.010403 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.010363 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-sys-fs\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.010488 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.010431 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jz5\" (UniqueName: \"kubernetes.io/projected/6e91d78e-4060-4063-86ac-5d7be516f269-kube-api-access-s5jz5\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:10:53.010488 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.010457 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 19:10:53.010488 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.010467 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-var-lib-openvswitch\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.010599 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.010491 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-run-netns\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.010599 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.010516 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-etc-kubernetes\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.010599 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.010542 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdvw\" (UniqueName: \"kubernetes.io/projected/a6bb4d98-4bbc-4c23-82ad-351fdd921751-kube-api-access-vzdvw\") pod \"node-ca-gndd8\" (UID: \"a6bb4d98-4bbc-4c23-82ad-351fdd921751\") " pod="openshift-image-registry/node-ca-gndd8" Jul 14 19:10:53.010599 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.010563 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-lib-modules\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.010599 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.010584 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-socket-dir\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.010973 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.010935 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-systemd-units\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.010999 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-hostroot\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011032 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-run-multus-certs\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011108 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl7jd\" (UniqueName: \"kubernetes.io/projected/8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f-kube-api-access-gl7jd\") pod \"node-resolver-vq5gr\" (UID: \"8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f\") " pod="openshift-dns/node-resolver-vq5gr" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011163 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6bb4d98-4bbc-4c23-82ad-351fdd921751-serviceca\") pod \"node-ca-gndd8\" (UID: \"a6bb4d98-4bbc-4c23-82ad-351fdd921751\") " pod="openshift-image-registry/node-ca-gndd8" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011203 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6cedffc7-b943-486f-9322-73a3b01c7d93-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011241 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/466721f5-d454-479b-97b6-9a8aad590b61-agent-certs\") pod \"konnectivity-agent-88fh2\" (UID: \"466721f5-d454-479b-97b6-9a8aad590b61\") " pod="kube-system/konnectivity-agent-88fh2" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011271 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-run-openvswitch\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011322 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-var-lib-kubelet\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011349 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-kubernetes\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011379 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-multus-socket-dir-parent\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011443 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-var-lib-cni-bin\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011473 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4f9j\" (UniqueName: \"kubernetes.io/projected/1a4a1223-587e-4b8e-b0ca-03a67c290d63-kube-api-access-h4f9j\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011505 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-registration-dir\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011528 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-slash\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011557 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cedffc7-b943-486f-9322-73a3b01c7d93-os-release\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.013253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011584 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-run\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011645 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-host\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011674 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011704 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-run-netns\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011732 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-node-log\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011756 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0286c27-b1d6-437f-90ff-5ef44af9d733-env-overrides\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011790 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-multus-conf-dir\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011820 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cedffc7-b943-486f-9322-73a3b01c7d93-system-cni-dir\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011850 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cedffc7-b943-486f-9322-73a3b01c7d93-cnibin\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011887 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cedffc7-b943-486f-9322-73a3b01c7d93-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011911 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-sysctl-conf\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.011938 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-systemd\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012033 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012087 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f-hosts-file\") pod \"node-resolver-vq5gr\" (UID: \"8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f\") " pod="openshift-dns/node-resolver-vq5gr" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9plp6\" (UniqueName: \"kubernetes.io/projected/6cedffc7-b943-486f-9322-73a3b01c7d93-kube-api-access-9plp6\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012142 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-run-k8s-cni-cncf-io\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.013970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012172 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-run-ovn-kubernetes\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012201 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-cni-bin\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012233 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-system-cni-dir\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012260 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-os-release\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012283 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7psb6\" (UniqueName: \"kubernetes.io/projected/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-kube-api-access-7psb6\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012315 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a4a1223-587e-4b8e-b0ca-03a67c290d63-tmp\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012344 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jttkn\" (UniqueName: \"kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn\") pod \"network-check-target-h5x8l\" (UID: \"dfb34251-8e7e-4321-9a87-0dd7af1a459c\") " pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012373 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0db1df7d-77b1-483f-92bb-8c9219e3b4e4-host-slash\") pod \"iptables-alerter-l65x7\" (UID: \"0db1df7d-77b1-483f-92bb-8c9219e3b4e4\") " pod="openshift-network-operator/iptables-alerter-l65x7" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012405 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb3e121f-d03c-496e-8aad-591b078ec990-dbus\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012434 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-log-socket\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012472 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012498 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0286c27-b1d6-437f-90ff-5ef44af9d733-ovnkube-config\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012524 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6bb4d98-4bbc-4c23-82ad-351fdd921751-host\") pod \"node-ca-gndd8\" (UID: \"a6bb4d98-4bbc-4c23-82ad-351fdd921751\") " pod="openshift-image-registry/node-ca-gndd8" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012552 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkggr\" (UniqueName: \"kubernetes.io/projected/0db1df7d-77b1-483f-92bb-8c9219e3b4e4-kube-api-access-hkggr\") pod \"iptables-alerter-l65x7\" (UID: \"0db1df7d-77b1-483f-92bb-8c9219e3b4e4\") " pod="openshift-network-operator/iptables-alerter-l65x7" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012580 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-device-dir\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012605 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-run-systemd\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.014499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012639 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6cedffc7-b943-486f-9322-73a3b01c7d93-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012779 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/466721f5-d454-479b-97b6-9a8aad590b61-konnectivity-ca\") pod \"konnectivity-agent-88fh2\" (UID: \"466721f5-d454-479b-97b6-9a8aad590b61\") " pod="kube-system/konnectivity-agent-88fh2" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012810 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-cni-netd\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012839 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-var-lib-cni-multus\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012868 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cedffc7-b943-486f-9322-73a3b01c7d93-cni-binary-copy\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012897 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-modprobe-d\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012925 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-sysctl-d\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.012953 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-kubelet\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013009 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-run-ovn\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013103 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0286c27-b1d6-437f-90ff-5ef44af9d733-ovnkube-script-lib\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013252 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-multus-daemon-config\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013303 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-etc-selinux\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013336 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wsxd\" (UniqueName: \"kubernetes.io/projected/67d28d9d-2b96-471c-b377-a2f7fa44b4df-kube-api-access-9wsxd\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013371 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0286c27-b1d6-437f-90ff-5ef44af9d733-ovn-node-metrics-cert\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013395 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-sys\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013434 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-var-lib-kubelet\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.015103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013472 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb3e121f-d03c-496e-8aad-591b078ec990-kubelet-config\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:53.015644 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013528 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-cnibin\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.015644 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013568 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-sysconfig\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.015644 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013679 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0db1df7d-77b1-483f-92bb-8c9219e3b4e4-iptables-alerter-script\") pod \"iptables-alerter-l65x7\" (UID: \"0db1df7d-77b1-483f-92bb-8c9219e3b4e4\") " pod="openshift-network-operator/iptables-alerter-l65x7" Jul 14 19:10:53.015644 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013727 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-etc-openvswitch\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.015644 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013790 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxhf\" (UniqueName: \"kubernetes.io/projected/b0286c27-b1d6-437f-90ff-5ef44af9d733-kube-api-access-tmxhf\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.015644 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013839 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-multus-cni-dir\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.015644 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.013886 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-cni-binary-copy\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.015644 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.014022 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f-tmp-dir\") pod \"node-resolver-vq5gr\" (UID: \"8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f\") " pod="openshift-dns/node-resolver-vq5gr" Jul 14 19:10:53.034552 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.034526 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8rs5g" Jul 14 19:10:53.040098 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.040073 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8rs5g" Jul 14 19:10:53.114626 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114599 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cedffc7-b943-486f-9322-73a3b01c7d93-os-release\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.114766 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114630 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-run\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.114766 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114647 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-host\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.114766 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114667 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:10:53.114766 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114682 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-run-netns\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.114766 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-node-log\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.114766 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114713 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-run\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.114766 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114723 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0286c27-b1d6-437f-90ff-5ef44af9d733-env-overrides\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.114766 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114731 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-host\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.114766 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114731 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cedffc7-b943-486f-9322-73a3b01c7d93-os-release\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.114766 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114759 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-node-log\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.114766 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114740 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-multus-conf-dir\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114790 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cedffc7-b943-486f-9322-73a3b01c7d93-system-cni-dir\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.114797 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114808 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cedffc7-b943-486f-9322-73a3b01c7d93-cnibin\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114825 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cedffc7-b943-486f-9322-73a3b01c7d93-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114833 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cedffc7-b943-486f-9322-73a3b01c7d93-system-cni-dir\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114791 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-multus-conf-dir\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114868 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cedffc7-b943-486f-9322-73a3b01c7d93-cnibin\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114839 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-sysctl-conf\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.114878 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs podName:6e91d78e-4060-4063-86ac-5d7be516f269 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:53.614856741 +0000 UTC m=+2.979927107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs") pod "network-metrics-daemon-h7v99" (UID: "6e91d78e-4060-4063-86ac-5d7be516f269") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114800 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-run-netns\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114904 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-systemd\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114923 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114940 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f-hosts-file\") pod \"node-resolver-vq5gr\" (UID: \"8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f\") " pod="openshift-dns/node-resolver-vq5gr" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114975 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-sysctl-conf\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.114981 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9plp6\" (UniqueName: \"kubernetes.io/projected/6cedffc7-b943-486f-9322-73a3b01c7d93-kube-api-access-9plp6\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115007 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-run-k8s-cni-cncf-io\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.115253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115014 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-systemd\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115029 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f-hosts-file\") pod \"node-resolver-vq5gr\" (UID: \"8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f\") " pod="openshift-dns/node-resolver-vq5gr" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115030 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-run-ovn-kubernetes\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115087 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-run-ovn-kubernetes\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.115088 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115124 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-run-k8s-cni-cncf-io\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.115140 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret podName:eb3e121f-d03c-496e-8aad-591b078ec990 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:53.615123409 +0000 UTC m=+2.980193755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret") pod "global-pull-secret-syncer-t6nzh" (UID: "eb3e121f-d03c-496e-8aad-591b078ec990") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115090 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-cni-bin\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115171 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-cni-bin\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115174 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-system-cni-dir\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115201 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-os-release\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115225 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7psb6\" (UniqueName: \"kubernetes.io/projected/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-kube-api-access-7psb6\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115243 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0286c27-b1d6-437f-90ff-5ef44af9d733-env-overrides\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115290 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-system-cni-dir\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115248 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a4a1223-587e-4b8e-b0ca-03a67c290d63-tmp\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115344 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttkn\" (UniqueName: \"kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn\") pod \"network-check-target-h5x8l\" (UID: \"dfb34251-8e7e-4321-9a87-0dd7af1a459c\") " pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115370 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0db1df7d-77b1-483f-92bb-8c9219e3b4e4-host-slash\") pod \"iptables-alerter-l65x7\" (UID: \"0db1df7d-77b1-483f-92bb-8c9219e3b4e4\") " pod="openshift-network-operator/iptables-alerter-l65x7" Jul 14 19:10:53.115894 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115374 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-os-release\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115412 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0db1df7d-77b1-483f-92bb-8c9219e3b4e4-host-slash\") pod \"iptables-alerter-l65x7\" (UID: \"0db1df7d-77b1-483f-92bb-8c9219e3b4e4\") " pod="openshift-network-operator/iptables-alerter-l65x7" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115477 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb3e121f-d03c-496e-8aad-591b078ec990-dbus\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115496 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115535 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-log-socket\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115551 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cedffc7-b943-486f-9322-73a3b01c7d93-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115505 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-log-socket\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115596 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb3e121f-d03c-496e-8aad-591b078ec990-dbus\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115605 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115635 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0286c27-b1d6-437f-90ff-5ef44af9d733-ovnkube-config\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6bb4d98-4bbc-4c23-82ad-351fdd921751-host\") pod \"node-ca-gndd8\" (UID: \"a6bb4d98-4bbc-4c23-82ad-351fdd921751\") " pod="openshift-image-registry/node-ca-gndd8" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115689 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkggr\" (UniqueName: \"kubernetes.io/projected/0db1df7d-77b1-483f-92bb-8c9219e3b4e4-kube-api-access-hkggr\") pod \"iptables-alerter-l65x7\" (UID: \"0db1df7d-77b1-483f-92bb-8c9219e3b4e4\") " pod="openshift-network-operator/iptables-alerter-l65x7" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115704 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115716 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-device-dir\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115750 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6bb4d98-4bbc-4c23-82ad-351fdd921751-host\") pod \"node-ca-gndd8\" (UID: \"a6bb4d98-4bbc-4c23-82ad-351fdd921751\") " pod="openshift-image-registry/node-ca-gndd8" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115776 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-device-dir\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115848 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-run-systemd\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.116680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115877 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6cedffc7-b943-486f-9322-73a3b01c7d93-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115902 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-run-systemd\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.115944 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/466721f5-d454-479b-97b6-9a8aad590b61-konnectivity-ca\") pod \"konnectivity-agent-88fh2\" (UID: \"466721f5-d454-479b-97b6-9a8aad590b61\") " pod="kube-system/konnectivity-agent-88fh2" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-cni-netd\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116086 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-var-lib-cni-multus\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116113 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cedffc7-b943-486f-9322-73a3b01c7d93-cni-binary-copy\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116136 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-modprobe-d\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116153 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-cni-netd\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116160 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-sysctl-d\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116207 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-kubelet\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116231 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-run-ovn\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116236 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0286c27-b1d6-437f-90ff-5ef44af9d733-ovnkube-config\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116254 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0286c27-b1d6-437f-90ff-5ef44af9d733-ovnkube-script-lib\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116280 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-sysctl-d\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116295 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-multus-daemon-config\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116321 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-etc-selinux\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116331 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-var-lib-cni-multus\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.117473 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116346 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wsxd\" (UniqueName: \"kubernetes.io/projected/67d28d9d-2b96-471c-b377-a2f7fa44b4df-kube-api-access-9wsxd\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116371 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0286c27-b1d6-437f-90ff-5ef44af9d733-ovn-node-metrics-cert\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116393 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-sys\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116404 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-modprobe-d\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116415 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-var-lib-kubelet\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116443 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb3e121f-d03c-496e-8aad-591b078ec990-kubelet-config\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116445 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-run-ovn\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116476 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-cnibin\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116502 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-sysconfig\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116528 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0db1df7d-77b1-483f-92bb-8c9219e3b4e4-iptables-alerter-script\") pod \"iptables-alerter-l65x7\" (UID: \"0db1df7d-77b1-483f-92bb-8c9219e3b4e4\") " pod="openshift-network-operator/iptables-alerter-l65x7" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116554 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-etc-openvswitch\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116579 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxhf\" (UniqueName: \"kubernetes.io/projected/b0286c27-b1d6-437f-90ff-5ef44af9d733-kube-api-access-tmxhf\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116587 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6cedffc7-b943-486f-9322-73a3b01c7d93-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116591 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/466721f5-d454-479b-97b6-9a8aad590b61-konnectivity-ca\") pod \"konnectivity-agent-88fh2\" (UID: \"466721f5-d454-479b-97b6-9a8aad590b61\") " pod="kube-system/konnectivity-agent-88fh2" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116601 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-multus-cni-dir\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116627 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-cni-binary-copy\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116651 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-sysconfig\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116650 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f-tmp-dir\") pod \"node-resolver-vq5gr\" (UID: \"8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f\") " pod="openshift-dns/node-resolver-vq5gr" Jul 14 19:10:53.118291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116688 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-tuned\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116714 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116723 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-var-lib-kubelet\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116297 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-kubelet\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116739 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-sys-fs\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116765 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jz5\" (UniqueName: \"kubernetes.io/projected/6e91d78e-4060-4063-86ac-5d7be516f269-kube-api-access-s5jz5\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116776 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb3e121f-d03c-496e-8aad-591b078ec990-kubelet-config\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116789 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-var-lib-openvswitch\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116815 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-run-netns\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116825 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cedffc7-b943-486f-9322-73a3b01c7d93-cni-binary-copy\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116843 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-etc-kubernetes\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116870 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdvw\" (UniqueName: \"kubernetes.io/projected/a6bb4d98-4bbc-4c23-82ad-351fdd921751-kube-api-access-vzdvw\") pod \"node-ca-gndd8\" (UID: \"a6bb4d98-4bbc-4c23-82ad-351fdd921751\") " pod="openshift-image-registry/node-ca-gndd8" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116884 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-sys-fs\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116895 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-lib-modules\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116925 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-socket-dir\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116949 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-systemd-units\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116972 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-hostroot\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.119185 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116999 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-run-multus-certs\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117014 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f-tmp-dir\") pod \"node-resolver-vq5gr\" (UID: \"8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f\") " pod="openshift-dns/node-resolver-vq5gr" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117022 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl7jd\" (UniqueName: \"kubernetes.io/projected/8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f-kube-api-access-gl7jd\") pod \"node-resolver-vq5gr\" (UID: \"8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f\") " pod="openshift-dns/node-resolver-vq5gr" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117073 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6bb4d98-4bbc-4c23-82ad-351fdd921751-serviceca\") pod \"node-ca-gndd8\" (UID: \"a6bb4d98-4bbc-4c23-82ad-351fdd921751\") " pod="openshift-image-registry/node-ca-gndd8" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116503 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-etc-selinux\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117100 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6cedffc7-b943-486f-9322-73a3b01c7d93-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117130 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/466721f5-d454-479b-97b6-9a8aad590b61-agent-certs\") pod \"konnectivity-agent-88fh2\" (UID: \"466721f5-d454-479b-97b6-9a8aad590b61\") " pod="kube-system/konnectivity-agent-88fh2" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117156 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-run-openvswitch\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-var-lib-kubelet\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-kubernetes\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117253 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-multus-socket-dir-parent\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117269 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0286c27-b1d6-437f-90ff-5ef44af9d733-ovnkube-script-lib\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117279 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-var-lib-cni-bin\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117307 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4f9j\" (UniqueName: \"kubernetes.io/projected/1a4a1223-587e-4b8e-b0ca-03a67c290d63-kube-api-access-h4f9j\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117335 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-registration-dir\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117359 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-slash\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117428 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.119752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117449 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-host-slash\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117481 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-sys\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117587 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-lib-modules\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117625 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-var-lib-openvswitch\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117663 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-run-netns\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117703 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-etc-kubernetes\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117736 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-multus-daemon-config\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117770 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-socket-dir\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117776 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-systemd-units\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.116927 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-cnibin\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117796 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-var-lib-kubelet\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117818 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-hostroot\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117852 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-etc-openvswitch\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117923 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-kubernetes\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117936 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-multus-cni-dir\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.117989 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67d28d9d-2b96-471c-b377-a2f7fa44b4df-registration-dir\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.118028 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-run-multus-certs\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.118054 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-multus-socket-dir-parent\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.120287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.118089 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-host-var-lib-cni-bin\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.120770 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.118159 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0286c27-b1d6-437f-90ff-5ef44af9d733-run-openvswitch\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.120770 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.118279 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6bb4d98-4bbc-4c23-82ad-351fdd921751-serviceca\") pod \"node-ca-gndd8\" (UID: \"a6bb4d98-4bbc-4c23-82ad-351fdd921751\") " pod="openshift-image-registry/node-ca-gndd8" Jul 14 19:10:53.120770 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.118529 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-cni-binary-copy\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.120770 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.118631 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6cedffc7-b943-486f-9322-73a3b01c7d93-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.120770 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.118808 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0db1df7d-77b1-483f-92bb-8c9219e3b4e4-iptables-alerter-script\") pod \"iptables-alerter-l65x7\" (UID: \"0db1df7d-77b1-483f-92bb-8c9219e3b4e4\") " pod="openshift-network-operator/iptables-alerter-l65x7" Jul 14 19:10:53.120770 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.118858 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a4a1223-587e-4b8e-b0ca-03a67c290d63-tmp\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.120770 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.119197 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1a4a1223-587e-4b8e-b0ca-03a67c290d63-etc-tuned\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.120770 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.119716 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0286c27-b1d6-437f-90ff-5ef44af9d733-ovn-node-metrics-cert\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.120770 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.120568 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/466721f5-d454-479b-97b6-9a8aad590b61-agent-certs\") pod \"konnectivity-agent-88fh2\" (UID: \"466721f5-d454-479b-97b6-9a8aad590b61\") " pod="kube-system/konnectivity-agent-88fh2" Jul 14 19:10:53.123256 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.123234 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9plp6\" (UniqueName: \"kubernetes.io/projected/6cedffc7-b943-486f-9322-73a3b01c7d93-kube-api-access-9plp6\") pod \"multus-additional-cni-plugins-tbjqz\" (UID: \"6cedffc7-b943-486f-9322-73a3b01c7d93\") " pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.123359 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.123270 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7psb6\" (UniqueName: \"kubernetes.io/projected/3dcd3c48-d0c0-4d6c-9681-af171f9a3567-kube-api-access-7psb6\") pod \"multus-dwftk\" (UID: \"3dcd3c48-d0c0-4d6c-9681-af171f9a3567\") " pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.127219 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.127198 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:10:53.127219 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.127221 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:10:53.127407 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.127235 2572 projected.go:196] Error preparing data for projected volume kube-api-access-jttkn for pod openshift-network-diagnostics/network-check-target-h5x8l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:53.127407 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.127299 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn podName:dfb34251-8e7e-4321-9a87-0dd7af1a459c nodeName:}" failed. No retries permitted until 2026-07-14 19:10:53.627281955 +0000 UTC m=+2.992352293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jttkn" (UniqueName: "kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn") pod "network-check-target-h5x8l" (UID: "dfb34251-8e7e-4321-9a87-0dd7af1a459c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:53.128887 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.128837 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkggr\" (UniqueName: \"kubernetes.io/projected/0db1df7d-77b1-483f-92bb-8c9219e3b4e4-kube-api-access-hkggr\") pod \"iptables-alerter-l65x7\" (UID: \"0db1df7d-77b1-483f-92bb-8c9219e3b4e4\") " pod="openshift-network-operator/iptables-alerter-l65x7" Jul 14 19:10:53.129541 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.128966 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wsxd\" (UniqueName: \"kubernetes.io/projected/67d28d9d-2b96-471c-b377-a2f7fa44b4df-kube-api-access-9wsxd\") pod \"aws-ebs-csi-driver-node-l5q2v\" (UID: \"67d28d9d-2b96-471c-b377-a2f7fa44b4df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.130973 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.130944 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl7jd\" (UniqueName: \"kubernetes.io/projected/8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f-kube-api-access-gl7jd\") pod \"node-resolver-vq5gr\" (UID: \"8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f\") " pod="openshift-dns/node-resolver-vq5gr" Jul 14 19:10:53.131213 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.131191 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4f9j\" (UniqueName: \"kubernetes.io/projected/1a4a1223-587e-4b8e-b0ca-03a67c290d63-kube-api-access-h4f9j\") pod \"tuned-p7l7s\" (UID: \"1a4a1223-587e-4b8e-b0ca-03a67c290d63\") " pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.131213 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.131203 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxhf\" (UniqueName: \"kubernetes.io/projected/b0286c27-b1d6-437f-90ff-5ef44af9d733-kube-api-access-tmxhf\") pod \"ovnkube-node-xw4dp\" (UID: \"b0286c27-b1d6-437f-90ff-5ef44af9d733\") " pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.132326 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.132303 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jz5\" (UniqueName: \"kubernetes.io/projected/6e91d78e-4060-4063-86ac-5d7be516f269-kube-api-access-s5jz5\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:10:53.133486 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.133465 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdvw\" (UniqueName: \"kubernetes.io/projected/a6bb4d98-4bbc-4c23-82ad-351fdd921751-kube-api-access-vzdvw\") pod \"node-ca-gndd8\" (UID: \"a6bb4d98-4bbc-4c23-82ad-351fdd921751\") " pod="openshift-image-registry/node-ca-gndd8" Jul 14 19:10:53.158888 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.158865 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gndd8" Jul 14 19:10:53.163879 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.163860 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tbjqz" Jul 14 19:10:53.317144 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.317063 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-88fh2" Jul 14 19:10:53.331086 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.331028 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" Jul 14 19:10:53.336806 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.336779 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l65x7" Jul 14 19:10:53.363377 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.363342 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:10:53.386265 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.386239 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" Jul 14 19:10:53.389652 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.389636 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dwftk" Jul 14 19:10:53.418762 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.418736 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:10:53.419248 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.419229 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vq5gr" Jul 14 19:10:53.452581 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:53.452478 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8debbe_5a4a_4bfc_bc3f_1ce3aa17f96f.slice/crio-d4496d983f5f2fd9bdba7af0519ff17636d4e527460e06eede7013c69a719007 WatchSource:0}: Error finding container d4496d983f5f2fd9bdba7af0519ff17636d4e527460e06eede7013c69a719007: Status 404 returned error can't find the container with id d4496d983f5f2fd9bdba7af0519ff17636d4e527460e06eede7013c69a719007 Jul 14 19:10:53.453054 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:53.453021 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fcd049ac89594ddba0d96bd90cd93b.slice/crio-83f9cd29da099f08fcbbe352bbc9fe781c2dfbb6c86448a04564581f3acbdb56 WatchSource:0}: Error finding container 83f9cd29da099f08fcbbe352bbc9fe781c2dfbb6c86448a04564581f3acbdb56: Status 404 returned error can't find the container with id 83f9cd29da099f08fcbbe352bbc9fe781c2dfbb6c86448a04564581f3acbdb56 Jul 14 19:10:53.454678 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:53.454659 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6bb4d98_4bbc_4c23_82ad_351fdd921751.slice/crio-24e41cb50a721d2f7d397aba05af25e5697e5c2f3959ccdd26395001436c0312 WatchSource:0}: Error finding container 24e41cb50a721d2f7d397aba05af25e5697e5c2f3959ccdd26395001436c0312: Status 404 returned error can't find the container with id 24e41cb50a721d2f7d397aba05af25e5697e5c2f3959ccdd26395001436c0312 Jul 14 19:10:53.457110 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.457096 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:10:53.530235 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.530195 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:10:53.620154 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.620060 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:10:53.620154 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.620098 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:53.620352 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.620208 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:53.620352 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.620220 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:53.620352 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.620274 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret podName:eb3e121f-d03c-496e-8aad-591b078ec990 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:54.620261573 +0000 UTC m=+3.985331911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret") pod "global-pull-secret-syncer-t6nzh" (UID: "eb3e121f-d03c-496e-8aad-591b078ec990") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:53.620352 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.620294 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs podName:6e91d78e-4060-4063-86ac-5d7be516f269 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:54.6202869 +0000 UTC m=+3.985357238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs") pod "network-metrics-daemon-h7v99" (UID: "6e91d78e-4060-4063-86ac-5d7be516f269") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:53.721422 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:53.721391 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttkn\" (UniqueName: \"kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn\") pod \"network-check-target-h5x8l\" (UID: \"dfb34251-8e7e-4321-9a87-0dd7af1a459c\") " pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:10:53.721607 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.721526 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:10:53.721607 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.721544 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:10:53.721607 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.721552 2572 projected.go:196] Error preparing data for projected volume kube-api-access-jttkn for pod openshift-network-diagnostics/network-check-target-h5x8l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:53.721607 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:53.721604 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn podName:dfb34251-8e7e-4321-9a87-0dd7af1a459c nodeName:}" failed. No retries permitted until 2026-07-14 19:10:54.721589138 +0000 UTC m=+4.086659491 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jttkn" (UniqueName: "kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn") pod "network-check-target-h5x8l" (UID: "dfb34251-8e7e-4321-9a87-0dd7af1a459c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:53.748195 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:53.748165 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4a1223_587e_4b8e_b0ca_03a67c290d63.slice/crio-f18c52b2ac9b590baea12f64116527ca03e2e47bb72d89d1d8067884ac038c4e WatchSource:0}: Error finding container f18c52b2ac9b590baea12f64116527ca03e2e47bb72d89d1d8067884ac038c4e: Status 404 returned error can't find the container with id f18c52b2ac9b590baea12f64116527ca03e2e47bb72d89d1d8067884ac038c4e Jul 14 19:10:53.780123 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:53.780095 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466721f5_d454_479b_97b6_9a8aad590b61.slice/crio-ca2ca9e71dc3b5d712202f9d7bd05881adb1201e05aca32b6bc7c4f6a2f94be7 WatchSource:0}: Error finding container ca2ca9e71dc3b5d712202f9d7bd05881adb1201e05aca32b6bc7c4f6a2f94be7: Status 404 returned error can't find the container with id ca2ca9e71dc3b5d712202f9d7bd05881adb1201e05aca32b6bc7c4f6a2f94be7 Jul 14 19:10:53.801259 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:53.801227 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0286c27_b1d6_437f_90ff_5ef44af9d733.slice/crio-a6cae5e5c7e029d7304e1d66158b37ab06e4966a89f7285a3f1d04000e0dbdc4 WatchSource:0}: Error finding container a6cae5e5c7e029d7304e1d66158b37ab06e4966a89f7285a3f1d04000e0dbdc4: Status 404 returned error can't find the container with id a6cae5e5c7e029d7304e1d66158b37ab06e4966a89f7285a3f1d04000e0dbdc4 Jul 14 19:10:54.041982 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:54.041900 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 19:05:53 +0000 UTC" deadline="2028-03-14 18:33:06.398089557 +0000 UTC" Jul 14 19:10:54.041982 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:54.041980 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14615h22m12.356113967s" Jul 14 19:10:54.189122 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:54.189063 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" event={"ID":"b0286c27-b1d6-437f-90ff-5ef44af9d733","Type":"ContainerStarted","Data":"a6cae5e5c7e029d7304e1d66158b37ab06e4966a89f7285a3f1d04000e0dbdc4"} Jul 14 19:10:54.191161 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:54.191131 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-88fh2" event={"ID":"466721f5-d454-479b-97b6-9a8aad590b61","Type":"ContainerStarted","Data":"ca2ca9e71dc3b5d712202f9d7bd05881adb1201e05aca32b6bc7c4f6a2f94be7"} Jul 14 19:10:54.193277 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:54.193249 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" event={"ID":"1a4a1223-587e-4b8e-b0ca-03a67c290d63","Type":"ContainerStarted","Data":"f18c52b2ac9b590baea12f64116527ca03e2e47bb72d89d1d8067884ac038c4e"} Jul 14 19:10:54.195036 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:54.194989 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gndd8" event={"ID":"a6bb4d98-4bbc-4c23-82ad-351fdd921751","Type":"ContainerStarted","Data":"24e41cb50a721d2f7d397aba05af25e5697e5c2f3959ccdd26395001436c0312"} Jul 14 19:10:54.196534 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:54.196507 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal" event={"ID":"71fcd049ac89594ddba0d96bd90cd93b","Type":"ContainerStarted","Data":"83f9cd29da099f08fcbbe352bbc9fe781c2dfbb6c86448a04564581f3acbdb56"} Jul 14 19:10:54.198081 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:54.198007 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vq5gr" event={"ID":"8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f","Type":"ContainerStarted","Data":"d4496d983f5f2fd9bdba7af0519ff17636d4e527460e06eede7013c69a719007"} Jul 14 19:10:54.629467 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:54.629430 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:10:54.629691 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:54.629484 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:54.629691 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:54.629629 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:54.629691 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:54.629687 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret podName:eb3e121f-d03c-496e-8aad-591b078ec990 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:56.629669048 +0000 UTC m=+5.994739402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret") pod "global-pull-secret-syncer-t6nzh" (UID: "eb3e121f-d03c-496e-8aad-591b078ec990") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:54.630115 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:54.630079 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:54.630201 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:54.630126 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs podName:6e91d78e-4060-4063-86ac-5d7be516f269 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:56.630111428 +0000 UTC m=+5.995181767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs") pod "network-metrics-daemon-h7v99" (UID: "6e91d78e-4060-4063-86ac-5d7be516f269") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:54.733443 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:54.733403 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttkn\" (UniqueName: \"kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn\") pod \"network-check-target-h5x8l\" (UID: \"dfb34251-8e7e-4321-9a87-0dd7af1a459c\") " pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:10:54.733640 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:54.733592 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:10:54.733640 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:54.733614 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:10:54.733640 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:54.733627 2572 projected.go:196] Error preparing data for projected volume kube-api-access-jttkn for pod openshift-network-diagnostics/network-check-target-h5x8l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:54.733800 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:54.733704 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn podName:dfb34251-8e7e-4321-9a87-0dd7af1a459c nodeName:}" failed. No retries permitted until 2026-07-14 19:10:56.73368549 +0000 UTC m=+6.098755846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jttkn" (UniqueName: "kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn") pod "network-check-target-h5x8l" (UID: "dfb34251-8e7e-4321-9a87-0dd7af1a459c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:55.042582 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:55.042473 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 19:05:53 +0000 UTC" deadline="2027-12-08 21:04:54.66169902 +0000 UTC" Jul 14 19:10:55.042582 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:55.042527 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12289h53m59.619175616s" Jul 14 19:10:55.183530 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:55.183497 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:55.183687 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:55.183552 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:10:55.183687 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:55.183497 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:10:55.183687 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:55.183623 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:10:55.183841 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:55.183691 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:10:55.183841 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:55.183774 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:10:56.146416 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:56.146382 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cedffc7_b943_486f_9322_73a3b01c7d93.slice/crio-ac0f85c38367681d5bc2e5f1832ac38fb18010287b34372e14943f847b926841 WatchSource:0}: Error finding container ac0f85c38367681d5bc2e5f1832ac38fb18010287b34372e14943f847b926841: Status 404 returned error can't find the container with id ac0f85c38367681d5bc2e5f1832ac38fb18010287b34372e14943f847b926841 Jul 14 19:10:56.205416 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:56.205370 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" event={"ID":"67d28d9d-2b96-471c-b377-a2f7fa44b4df","Type":"ContainerStarted","Data":"568a54b7033d1eb4971044ef9e8fcbb2fd8d3ee5fbcc05152a1e13d9588a7b00"} Jul 14 19:10:56.207011 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:56.206964 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbjqz" event={"ID":"6cedffc7-b943-486f-9322-73a3b01c7d93","Type":"ContainerStarted","Data":"ac0f85c38367681d5bc2e5f1832ac38fb18010287b34372e14943f847b926841"} Jul 14 19:10:56.244936 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:10:56.244892 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04179ee16bc51140974d528cab249f3.slice/crio-ee3cca2e2f1eaf51dd5ff2e6be732490a3277139199056eb21d18f0074b6870f WatchSource:0}: Error finding container ee3cca2e2f1eaf51dd5ff2e6be732490a3277139199056eb21d18f0074b6870f: Status 404 returned error can't find the container with id ee3cca2e2f1eaf51dd5ff2e6be732490a3277139199056eb21d18f0074b6870f Jul 14 19:10:56.647175 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:56.646847 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:10:56.647175 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:56.647152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:56.647401 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:56.647337 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:56.647457 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:56.647401 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret podName:eb3e121f-d03c-496e-8aad-591b078ec990 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:00.647382005 +0000 UTC m=+10.012452360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret") pod "global-pull-secret-syncer-t6nzh" (UID: "eb3e121f-d03c-496e-8aad-591b078ec990") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:56.647523 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:56.647469 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:56.647523 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:56.647503 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs podName:6e91d78e-4060-4063-86ac-5d7be516f269 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:00.647491745 +0000 UTC m=+10.012562088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs") pod "network-metrics-daemon-h7v99" (UID: "6e91d78e-4060-4063-86ac-5d7be516f269") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:56.748350 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:56.748309 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttkn\" (UniqueName: \"kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn\") pod \"network-check-target-h5x8l\" (UID: \"dfb34251-8e7e-4321-9a87-0dd7af1a459c\") " pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:10:56.748526 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:56.748480 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:10:56.748526 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:56.748500 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:10:56.748526 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:56.748512 2572 projected.go:196] Error preparing data for projected volume kube-api-access-jttkn for pod openshift-network-diagnostics/network-check-target-h5x8l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:56.748678 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:56.748571 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn podName:dfb34251-8e7e-4321-9a87-0dd7af1a459c nodeName:}" failed. No retries permitted until 2026-07-14 19:11:00.748552563 +0000 UTC m=+10.113622901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jttkn" (UniqueName: "kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn") pod "network-check-target-h5x8l" (UID: "dfb34251-8e7e-4321-9a87-0dd7af1a459c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:57.185354 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:57.185322 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:10:57.185952 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:57.185464 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:10:57.185952 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:57.185833 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:57.185952 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:57.185918 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:10:57.186235 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:57.185962 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:10:57.186235 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:57.186021 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:10:57.216992 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:57.216952 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vq5gr" event={"ID":"8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f","Type":"ContainerStarted","Data":"013fc3583cc759b55a16c8c5011d13a18b298470f85bfd2de8bcdae2b88b44b5"} Jul 14 19:10:57.220213 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:57.220179 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.ec2.internal" event={"ID":"b04179ee16bc51140974d528cab249f3","Type":"ContainerStarted","Data":"ee3cca2e2f1eaf51dd5ff2e6be732490a3277139199056eb21d18f0074b6870f"} Jul 14 19:10:57.222657 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:57.222628 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l65x7" event={"ID":"0db1df7d-77b1-483f-92bb-8c9219e3b4e4","Type":"ContainerStarted","Data":"0150a81384d87d4473ff1c54905e83ffc88a2cd434c544e5fa8bf9f74dc5ccd0"} Jul 14 19:10:57.225561 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:57.225529 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-88fh2" event={"ID":"466721f5-d454-479b-97b6-9a8aad590b61","Type":"ContainerStarted","Data":"3cf3adea730069924c766cd28d4d736e0d1ee303525897cdb5a991d16ea10097"} Jul 14 19:10:57.229163 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:57.229134 2572 generic.go:356] "Generic (PLEG): container finished" podID="71fcd049ac89594ddba0d96bd90cd93b" containerID="6ef052c0e6dff71a9115dbf3bd143852c27884d52393f2b057432ab1ab728ce0" exitCode=0 Jul 14 19:10:57.229314 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:57.229180 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal" event={"ID":"71fcd049ac89594ddba0d96bd90cd93b","Type":"ContainerDied","Data":"6ef052c0e6dff71a9115dbf3bd143852c27884d52393f2b057432ab1ab728ce0"} Jul 14 19:10:57.251866 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:57.251286 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-88fh2" podStartSLOduration=3.914467928 podStartE2EDuration="6.251263709s" totalImagesPullingTime="2.336795781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:10:57.25032399 +0000 UTC m=+6.615394355" watchObservedRunningTime="2026-07-14 19:10:57.251263709 +0000 UTC m=+6.616334072" Jul 14 19:10:57.251866 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:57.251569 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vq5gr" podStartSLOduration=3.593179836 podStartE2EDuration="6.251558664s" totalImagesPullingTime="2.658378828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:10:57.232962985 +0000 UTC m=+6.598033350" watchObservedRunningTime="2026-07-14 19:10:57.251558664 +0000 UTC m=+6.616629025" Jul 14 19:10:58.234069 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:58.233774 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l65x7" event={"ID":"0db1df7d-77b1-483f-92bb-8c9219e3b4e4","Type":"ContainerStarted","Data":"6888849b92f3bf6581d7efa0afff74fb4a499930b19f0f10dd8c80cd61bf2a9e"} Jul 14 19:10:58.237940 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:58.237872 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal" event={"ID":"71fcd049ac89594ddba0d96bd90cd93b","Type":"ContainerStarted","Data":"a82882fb68f770342ec095f7a12e662812be8f6229d8e99b432cf2ce2a03eff7"} Jul 14 19:10:58.250358 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:58.250197 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l65x7" podStartSLOduration=7.250175251 podStartE2EDuration="7.250175251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:10:58.249206448 +0000 UTC m=+7.614276810" watchObservedRunningTime="2026-07-14 19:10:58.250175251 +0000 UTC m=+7.615245618" Jul 14 19:10:58.264625 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:58.264180 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-70.ec2.internal" podStartSLOduration=6.26416473 podStartE2EDuration="6.26416473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:10:58.263782533 +0000 UTC m=+7.628852895" watchObservedRunningTime="2026-07-14 19:10:58.26416473 +0000 UTC m=+7.629235093" Jul 14 19:10:58.568661 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:58.568621 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-88fh2" Jul 14 19:10:58.569259 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:58.569234 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-88fh2" Jul 14 19:10:59.183664 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:59.183607 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:10:59.183857 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:59.183618 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:10:59.183857 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:59.183737 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:10:59.183857 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:59.183811 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:10:59.184009 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:10:59.183926 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:10:59.184075 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:10:59.184007 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:11:00.241666 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:00.241584 2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 19:11:00.683910 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:00.683825 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:00.683910 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:00.683872 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:00.684168 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:00.683964 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:11:00.684168 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:00.683977 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:11:00.684168 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:00.684030 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret podName:eb3e121f-d03c-496e-8aad-591b078ec990 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:08.684013287 +0000 UTC m=+18.049083629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret") pod "global-pull-secret-syncer-t6nzh" (UID: "eb3e121f-d03c-496e-8aad-591b078ec990") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:11:00.684168 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:00.684065 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs podName:6e91d78e-4060-4063-86ac-5d7be516f269 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:08.684055091 +0000 UTC m=+18.049125434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs") pod "network-metrics-daemon-h7v99" (UID: "6e91d78e-4060-4063-86ac-5d7be516f269") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:11:00.784596 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:00.784549 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttkn\" (UniqueName: \"kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn\") pod \"network-check-target-h5x8l\" (UID: \"dfb34251-8e7e-4321-9a87-0dd7af1a459c\") " pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:00.784766 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:00.784748 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:11:00.784856 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:00.784773 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:11:00.784856 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:00.784787 2572 projected.go:196] Error preparing data for projected volume kube-api-access-jttkn for pod openshift-network-diagnostics/network-check-target-h5x8l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:11:00.784856 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:00.784852 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn podName:dfb34251-8e7e-4321-9a87-0dd7af1a459c nodeName:}" failed. No retries permitted until 2026-07-14 19:11:08.784832854 +0000 UTC m=+18.149903199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jttkn" (UniqueName: "kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn") pod "network-check-target-h5x8l" (UID: "dfb34251-8e7e-4321-9a87-0dd7af1a459c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:11:01.184026 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:01.184000 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:01.184220 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:01.184098 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:01.184220 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:01.184213 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:01.184339 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:01.184229 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:11:01.184397 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:01.184327 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:11:01.184443 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:01.184424 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:11:03.183833 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:03.183564 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:03.184459 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:03.183564 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:03.184459 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:03.183949 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:11:03.184459 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:03.183984 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:11:03.184459 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:03.183594 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:03.184459 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:03.184092 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:11:05.182794 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:05.182754 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:05.182794 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:05.182790 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:05.183345 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:05.182754 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:05.183345 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:05.182892 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:11:05.183345 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:05.182960 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:11:05.183345 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:05.183108 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:11:07.182664 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:07.182624 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:07.183140 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:07.182624 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:07.183140 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:07.182754 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:11:07.183140 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:07.182848 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:07.183140 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:07.182843 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:11:07.183140 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:07.182968 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:11:07.943910 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:11:07.943881 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcd3c48_d0c0_4d6c_9681_af171f9a3567.slice/crio-93563c99034738a54f06c7cbef9dab3aa1fd5fe25db76639b8997190659ac323 WatchSource:0}: Error finding container 93563c99034738a54f06c7cbef9dab3aa1fd5fe25db76639b8997190659ac323: Status 404 returned error can't find the container with id 93563c99034738a54f06c7cbef9dab3aa1fd5fe25db76639b8997190659ac323 Jul 14 19:11:08.263075 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.262895 2572 generic.go:356] "Generic (PLEG): container finished" podID="6cedffc7-b943-486f-9322-73a3b01c7d93" containerID="9bc9f6e18c88b53d6f64c7e113f05ec2555bd797efbce918d184f555621ee450" exitCode=0 Jul 14 19:11:08.263781 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.262974 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbjqz" event={"ID":"6cedffc7-b943-486f-9322-73a3b01c7d93","Type":"ContainerDied","Data":"9bc9f6e18c88b53d6f64c7e113f05ec2555bd797efbce918d184f555621ee450"} Jul 14 19:11:08.264682 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.264654 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dwftk" event={"ID":"3dcd3c48-d0c0-4d6c-9681-af171f9a3567","Type":"ContainerStarted","Data":"93563c99034738a54f06c7cbef9dab3aa1fd5fe25db76639b8997190659ac323"} Jul 14 19:11:08.266680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.266663 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:11:08.266983 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.266963 2572 generic.go:356] "Generic (PLEG): container finished" podID="b0286c27-b1d6-437f-90ff-5ef44af9d733" containerID="9b1f6dcf5df7d6fdfffef79081025144fd59c2dcf2615fe565e295a4df785d16" exitCode=1 Jul 14 19:11:08.267118 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.267097 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" event={"ID":"b0286c27-b1d6-437f-90ff-5ef44af9d733","Type":"ContainerStarted","Data":"ec0de99c0fbd2b6ed7af46abaabff3a4774d7d0d80861c192cec92e85fb563d9"} Jul 14 19:11:08.267199 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.267185 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" event={"ID":"b0286c27-b1d6-437f-90ff-5ef44af9d733","Type":"ContainerDied","Data":"9b1f6dcf5df7d6fdfffef79081025144fd59c2dcf2615fe565e295a4df785d16"} Jul 14 19:11:08.267267 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.267243 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" event={"ID":"b0286c27-b1d6-437f-90ff-5ef44af9d733","Type":"ContainerStarted","Data":"92b970de6fb64003d27c54bdef1662194cc4d5ecf3f1013a51829d4087f1ccc7"} Jul 14 19:11:08.268504 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.268471 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" event={"ID":"1a4a1223-587e-4b8e-b0ca-03a67c290d63","Type":"ContainerStarted","Data":"ed06cd5f259b5f1dbd3ef600dd9489f75c29a1c9db8b588bcd5e3630629a9f94"} Jul 14 19:11:08.270036 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.270011 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gndd8" event={"ID":"a6bb4d98-4bbc-4c23-82ad-351fdd921751","Type":"ContainerStarted","Data":"de0ccf79165b7162d088d836a26ae8748a6bace4bbf16adaf8998e5d8fba1691"} Jul 14 19:11:08.271541 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.271521 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" event={"ID":"67d28d9d-2b96-471c-b377-a2f7fa44b4df","Type":"ContainerStarted","Data":"933080659f86f74a1e84cb8f88f951dac5eda712ed03919b4ca44f3c50d4a8ff"} Jul 14 19:11:08.272903 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.272880 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.ec2.internal" event={"ID":"b04179ee16bc51140974d528cab249f3","Type":"ContainerStarted","Data":"1567a0464c1d1a52e4feeae0526a2bb199b99258c8d19621c40a37d6425eaa76"} Jul 14 19:11:08.302727 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.302650 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p7l7s" podStartSLOduration=3.131937648 podStartE2EDuration="17.302633425s" totalImagesPullingTime="14.170695777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:08.302590225 +0000 UTC m=+17.667660602" watchObservedRunningTime="2026-07-14 19:11:08.302633425 +0000 UTC m=+17.667703786" Jul 14 19:11:08.317123 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.317078 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gndd8" podStartSLOduration=2.853690264 podStartE2EDuration="17.317062287s" totalImagesPullingTime="14.463372023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:08.316796067 +0000 UTC m=+17.681866429" watchObservedRunningTime="2026-07-14 19:11:08.317062287 +0000 UTC m=+17.682132645" Jul 14 19:11:08.350747 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.350693 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-70.ec2.internal" podStartSLOduration=16.350676385 podStartE2EDuration="16.350676385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:08.350496046 +0000 UTC m=+17.715566406" watchObservedRunningTime="2026-07-14 19:11:08.350676385 +0000 UTC m=+17.715746745" Jul 14 19:11:08.741954 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.741900 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:08.741954 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.741958 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:08.742225 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:08.742094 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:11:08.742225 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:08.742128 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:11:08.742225 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:08.742172 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs podName:6e91d78e-4060-4063-86ac-5d7be516f269 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:24.742151494 +0000 UTC m=+34.107221846 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs") pod "network-metrics-daemon-h7v99" (UID: "6e91d78e-4060-4063-86ac-5d7be516f269") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:11:08.742225 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:08.742195 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret podName:eb3e121f-d03c-496e-8aad-591b078ec990 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:24.742181896 +0000 UTC m=+34.107252259 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret") pod "global-pull-secret-syncer-t6nzh" (UID: "eb3e121f-d03c-496e-8aad-591b078ec990") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:11:08.843378 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:08.843308 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttkn\" (UniqueName: \"kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn\") pod \"network-check-target-h5x8l\" (UID: \"dfb34251-8e7e-4321-9a87-0dd7af1a459c\") " pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:08.843564 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:08.843486 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:11:08.843564 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:08.843516 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:11:08.843564 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:08.843529 2572 projected.go:196] Error preparing data for projected volume kube-api-access-jttkn for pod openshift-network-diagnostics/network-check-target-h5x8l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:11:08.843667 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:08.843591 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn podName:dfb34251-8e7e-4321-9a87-0dd7af1a459c nodeName:}" failed. No retries permitted until 2026-07-14 19:11:24.843575766 +0000 UTC m=+34.208646105 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jttkn" (UniqueName: "kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn") pod "network-check-target-h5x8l" (UID: "dfb34251-8e7e-4321-9a87-0dd7af1a459c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:11:09.021302 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:09.021277 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 19:11:09.070109 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:09.069971 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T19:11:09.021298202Z","UUID":"091c862d-556a-4df2-856d-42fa1a466418","Handler":null,"Name":"","Endpoint":""} Jul 14 19:11:09.071650 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:09.071621 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 19:11:09.071818 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:09.071663 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 19:11:09.182604 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:09.182570 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:09.182783 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:09.182618 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:09.182783 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:09.182702 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:11:09.182783 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:09.182751 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:11:09.182945 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:09.182816 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:09.182945 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:09.182926 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:11:09.278556 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:09.278474 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:11:09.279143 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:09.279010 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" event={"ID":"b0286c27-b1d6-437f-90ff-5ef44af9d733","Type":"ContainerStarted","Data":"d06fc3ee9076a6245289d21693c82a465820fa1663a7454bbae956d2da4a6e2d"} Jul 14 19:11:09.279143 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:09.279119 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" event={"ID":"b0286c27-b1d6-437f-90ff-5ef44af9d733","Type":"ContainerStarted","Data":"7db3317706b2598946c4b82b192b053e08161ae146b0c65a97d90cd14d155f9a"} Jul 14 19:11:09.279250 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:09.279173 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" event={"ID":"b0286c27-b1d6-437f-90ff-5ef44af9d733","Type":"ContainerStarted","Data":"dd8d70b280f0d9f713b093f06a95855fe82c28e68b2157134acf4573169358cc"} Jul 14 19:11:09.281393 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:09.281363 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" event={"ID":"67d28d9d-2b96-471c-b377-a2f7fa44b4df","Type":"ContainerStarted","Data":"8080906ff258b5a30921a57c6f43fc6cc9233054a421a2f1fbf5f72c5e224832"} Jul 14 19:11:10.286312 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:10.285992 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" event={"ID":"67d28d9d-2b96-471c-b377-a2f7fa44b4df","Type":"ContainerStarted","Data":"70a7d4da755810838cc4565de3c72c598b1b97e324279ab25873064bd16b624e"} Jul 14 19:11:10.305598 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:10.305541 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5q2v" podStartSLOduration=5.815870787 podStartE2EDuration="19.305519938s" totalImagesPullingTime="13.489649151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:10.304682649 +0000 UTC m=+19.669753011" watchObservedRunningTime="2026-07-14 19:11:10.305519938 +0000 UTC m=+19.670590297" Jul 14 19:11:11.183599 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:11.183563 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:11.183780 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:11.183692 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:11.183780 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:11.183706 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:11:11.183903 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:11.183797 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:11:11.183903 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:11.183875 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:11.184016 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:11.183947 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:11:11.292681 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:11.292650 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:11:11.293190 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:11.293076 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" event={"ID":"b0286c27-b1d6-437f-90ff-5ef44af9d733","Type":"ContainerStarted","Data":"a7e6d36f7b5b8da22157856f544efaa0c644de37bfa9a89c3df2abb4c1ea3358"} Jul 14 19:11:13.182792 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:13.182760 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:13.183273 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:13.182834 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:13.183273 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:13.182756 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:13.183273 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:13.182923 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:11:13.183273 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:13.183072 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:11:13.183273 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:13.183159 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:11:14.302345 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:14.302304 2572 generic.go:356] "Generic (PLEG): container finished" podID="6cedffc7-b943-486f-9322-73a3b01c7d93" containerID="f5fafec932c8d5333391de1f8db14498f454f194b3860f0e77418232268b3888" exitCode=0 Jul 14 19:11:14.302876 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:14.302384 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbjqz" event={"ID":"6cedffc7-b943-486f-9322-73a3b01c7d93","Type":"ContainerDied","Data":"f5fafec932c8d5333391de1f8db14498f454f194b3860f0e77418232268b3888"} Jul 14 19:11:14.306821 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:14.306802 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:11:14.307192 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:14.307165 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" event={"ID":"b0286c27-b1d6-437f-90ff-5ef44af9d733","Type":"ContainerStarted","Data":"deff5dd06dbab7c8a12d3516c13cbf24fe76f73c2ee37b26f643199e5ed00253"} Jul 14 19:11:14.307523 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:14.307503 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:11:14.307603 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:14.307579 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:11:14.307784 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:14.307767 2572 scope.go:117] "RemoveContainer" containerID="9b1f6dcf5df7d6fdfffef79081025144fd59c2dcf2615fe565e295a4df785d16" Jul 14 19:11:14.324888 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:14.324856 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:11:15.183465 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:15.183434 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:15.183628 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:15.183472 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:15.183628 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:15.183435 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:15.183628 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:15.183576 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:11:15.183752 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:15.183653 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:11:15.183789 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:15.183746 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:11:15.657658 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:15.657547 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t6nzh"] Jul 14 19:11:15.658099 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:15.657697 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:15.658099 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:15.657802 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:11:15.660578 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:15.660432 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h7v99"] Jul 14 19:11:15.660578 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:15.660555 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:15.660766 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:15.660662 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:11:15.661248 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:15.661209 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h5x8l"] Jul 14 19:11:15.661362 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:15.661308 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:15.661428 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:15.661407 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:11:17.187440 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:17.187198 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:17.187923 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:17.187211 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:17.187923 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:17.187556 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:11:17.187923 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:17.187236 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:17.187923 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:17.187641 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:11:17.187923 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:17.187680 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:11:18.321092 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:18.321063 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:11:18.321773 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:18.321469 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" event={"ID":"b0286c27-b1d6-437f-90ff-5ef44af9d733","Type":"ContainerStarted","Data":"fc1efc731b07b8ef20f0bf6d95513359726f908a8073b6c67194aa3a680b532d"} Jul 14 19:11:18.321773 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:18.321636 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:11:18.334174 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:18.334138 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" podUID="b0286c27-b1d6-437f-90ff-5ef44af9d733" containerName="ovnkube-controller" probeResult="failure" output="" Jul 14 19:11:18.335736 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:18.335715 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:11:18.349243 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:18.349196 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" podStartSLOduration=13.150884967 podStartE2EDuration="27.349179792s" totalImagesPullingTime="14.198294825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:18.349128106 +0000 UTC m=+27.714198469" watchObservedRunningTime="2026-07-14 19:11:18.349179792 +0000 UTC m=+27.714250155" Jul 14 19:11:19.183578 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:19.183542 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:19.183578 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:19.183562 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:19.183804 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:19.183572 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:19.183804 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:19.183671 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6nzh" podUID="eb3e121f-d03c-496e-8aad-591b078ec990" Jul 14 19:11:19.183804 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:19.183789 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:11:19.183959 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:19.183857 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5x8l" podUID="dfb34251-8e7e-4321-9a87-0dd7af1a459c" Jul 14 19:11:19.325708 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:19.325449 2572 generic.go:356] "Generic (PLEG): container finished" podID="6cedffc7-b943-486f-9322-73a3b01c7d93" containerID="9efe3f517ed0d506230f40583d39d46afaf67a06fc326604bad05203f6dcff7e" exitCode=0 Jul 14 19:11:19.326170 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:19.325532 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbjqz" event={"ID":"6cedffc7-b943-486f-9322-73a3b01c7d93","Type":"ContainerDied","Data":"9efe3f517ed0d506230f40583d39d46afaf67a06fc326604bad05203f6dcff7e"} Jul 14 19:11:19.327248 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:19.327220 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dwftk" event={"ID":"3dcd3c48-d0c0-4d6c-9681-af171f9a3567","Type":"ContainerStarted","Data":"d5c12676c5703c1c0ec0f088fce9d6c6cce670dfef715155c197c90509dd33a5"} Jul 14 19:11:19.336630 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:19.336607 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xw4dp" Jul 14 19:11:19.391348 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:19.391244 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dwftk" podStartSLOduration=17.904042318 podStartE2EDuration="28.391228592s" totalImagesPullingTime="10.487186274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:19.391192007 +0000 UTC m=+28.756262367" watchObservedRunningTime="2026-07-14 19:11:19.391228592 +0000 UTC m=+28.756298952" Jul 14 19:11:21.009677 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.009643 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-70.ec2.internal" event="NodeReady" Jul 14 19:11:21.010121 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.009813 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 19:11:21.044060 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.044012 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j"] Jul 14 19:11:21.058672 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.058643 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g"] Jul 14 19:11:21.058854 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.058780 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" Jul 14 19:11:21.061796 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.061616 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-tbvh6\"" Jul 14 19:11:21.062031 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.062011 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 14 19:11:21.062675 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.062624 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 14 19:11:21.063306 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.062850 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 14 19:11:21.063306 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.063130 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 14 19:11:21.074900 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.074844 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f44db5d46-9wzvr"] Jul 14 19:11:21.075081 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.075036 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.077591 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.077571 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 14 19:11:21.077713 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.077588 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 14 19:11:21.077713 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.077604 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 14 19:11:21.077713 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.077573 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 14 19:11:21.097390 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.097363 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh"] Jul 14 19:11:21.097539 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.097477 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.100023 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.099998 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-plsql\"" Jul 14 19:11:21.100175 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.100090 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 19:11:21.100175 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.100101 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 19:11:21.100460 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.100444 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 19:11:21.104711 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.104695 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 19:11:21.120029 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.119982 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:21.123026 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.123004 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j"] Jul 14 19:11:21.123157 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.123037 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4kwvb"] Jul 14 19:11:21.123209 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.123155 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:11:21.125587 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.125567 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 14 19:11:21.130582 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.130562 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghj7r\" (UniqueName: \"kubernetes.io/projected/3542f768-c658-451b-b911-b8956f9e8e39-kube-api-access-ghj7r\") pod \"managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j\" (UID: \"3542f768-c658-451b-b911-b8956f9e8e39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" Jul 14 19:11:21.130684 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.130629 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3542f768-c658-451b-b911-b8956f9e8e39-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j\" (UID: \"3542f768-c658-451b-b911-b8956f9e8e39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" Jul 14 19:11:21.140032 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.140008 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh"] Jul 14 19:11:21.140146 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.140057 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g"] Jul 14 19:11:21.140146 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.140076 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-95h2r"] Jul 14 19:11:21.140232 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.140163 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:21.142321 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.142300 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 19:11:21.142465 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.142316 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5x4l\"" Jul 14 19:11:21.142465 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.142394 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 19:11:21.150031 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.150011 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f44db5d46-9wzvr"] Jul 14 19:11:21.150133 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.150055 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-95h2r"] Jul 14 19:11:21.150133 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.150078 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4kwvb"] Jul 14 19:11:21.150133 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.150117 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-95h2r" Jul 14 19:11:21.152488 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.152473 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cxj8f\"" Jul 14 19:11:21.152577 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.152536 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 19:11:21.152577 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.152474 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 19:11:21.152661 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.152580 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 19:11:21.183426 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.183393 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:21.183600 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.183561 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:21.183673 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.183638 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:21.186105 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.186080 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 19:11:21.186245 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.186213 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 19:11:21.186309 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.186213 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwrqs\"" Jul 14 19:11:21.186309 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.186270 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xxpfz\"" Jul 14 19:11:21.186309 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.186283 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 19:11:21.186309 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.186293 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 19:11:21.217711 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.217662 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:21.231586 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.231555 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2c03a20-3487-4987-b514-d80fbb77da40-registry-certificates\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.231701 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.231599 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4b0f202-374a-4e71-97db-50a6422c7767-cert\") pod \"ingress-canary-95h2r\" (UID: \"b4b0f202-374a-4e71-97db-50a6422c7767\") " pod="openshift-ingress-canary/ingress-canary-95h2r" Jul 14 19:11:21.231701 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.231617 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-ca\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.231701 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.231660 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-hub\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.231795 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.231698 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.231795 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.231755 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7stcj\" (UniqueName: \"kubernetes.io/projected/005d066e-4e27-42af-8e47-e6691b93f148-kube-api-access-7stcj\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:21.231795 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.231785 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2c03a20-3487-4987-b514-d80fbb77da40-ca-trust-extracted\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.231885 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.231824 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghj7r\" (UniqueName: \"kubernetes.io/projected/3542f768-c658-451b-b911-b8956f9e8e39-kube-api-access-ghj7r\") pod \"managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j\" (UID: \"3542f768-c658-451b-b911-b8956f9e8e39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" Jul 14 19:11:21.231885 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.231849 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3542f768-c658-451b-b911-b8956f9e8e39-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j\" (UID: \"3542f768-c658-451b-b911-b8956f9e8e39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" Jul 14 19:11:21.231885 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.231873 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.232030 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232012 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4919425-84e6-4f07-821c-87d4af4e71af-tmp\") pod \"klusterlet-addon-workmgr-789b67ddd8-v7lrh\" (UID: \"e4919425-84e6-4f07-821c-87d4af4e71af\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:11:21.232140 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.232186 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232154 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2c03a20-3487-4987-b514-d80fbb77da40-image-registry-private-configuration\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.232186 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232177 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mdlj\" (UniqueName: \"kubernetes.io/projected/b4b0f202-374a-4e71-97db-50a6422c7767-kube-api-access-8mdlj\") pod \"ingress-canary-95h2r\" (UID: \"b4b0f202-374a-4e71-97db-50a6422c7767\") " pod="openshift-ingress-canary/ingress-canary-95h2r" Jul 14 19:11:21.232272 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232194 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.232272 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232211 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfkkm\" (UniqueName: \"kubernetes.io/projected/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-kube-api-access-kfkkm\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.232272 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232239 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e4919425-84e6-4f07-821c-87d4af4e71af-klusterlet-config\") pod \"klusterlet-addon-workmgr-789b67ddd8-v7lrh\" (UID: \"e4919425-84e6-4f07-821c-87d4af4e71af\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:11:21.232272 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232259 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2c03a20-3487-4987-b514-d80fbb77da40-trusted-ca\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.232416 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232275 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/005d066e-4e27-42af-8e47-e6691b93f148-config-volume\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:21.232416 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232296 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:21.232416 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232354 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2c03a20-3487-4987-b514-d80fbb77da40-installation-pull-secrets\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.232416 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232387 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4qqw\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-kube-api-access-r4qqw\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.232557 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232427 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-bound-sa-token\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.232557 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232449 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005d066e-4e27-42af-8e47-e6691b93f148-tmp-dir\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:21.232557 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.232472 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4kxq\" (UniqueName: \"kubernetes.io/projected/e4919425-84e6-4f07-821c-87d4af4e71af-kube-api-access-p4kxq\") pod \"klusterlet-addon-workmgr-789b67ddd8-v7lrh\" (UID: \"e4919425-84e6-4f07-821c-87d4af4e71af\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:11:21.236290 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.236271 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3542f768-c658-451b-b911-b8956f9e8e39-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j\" (UID: \"3542f768-c658-451b-b911-b8956f9e8e39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" Jul 14 19:11:21.240410 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.240388 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghj7r\" (UniqueName: \"kubernetes.io/projected/3542f768-c658-451b-b911-b8956f9e8e39-kube-api-access-ghj7r\") pod \"managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j\" (UID: \"3542f768-c658-451b-b911-b8956f9e8e39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" Jul 14 19:11:21.332813 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.332721 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7stcj\" (UniqueName: \"kubernetes.io/projected/005d066e-4e27-42af-8e47-e6691b93f148-kube-api-access-7stcj\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:21.332813 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.332758 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2c03a20-3487-4987-b514-d80fbb77da40-ca-trust-extracted\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.332813 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.332806 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.333126 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.332833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4919425-84e6-4f07-821c-87d4af4e71af-tmp\") pod \"klusterlet-addon-workmgr-789b67ddd8-v7lrh\" (UID: \"e4919425-84e6-4f07-821c-87d4af4e71af\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:11:21.333126 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.332879 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.333126 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.332901 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2c03a20-3487-4987-b514-d80fbb77da40-image-registry-private-configuration\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.333126 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.332922 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mdlj\" (UniqueName: \"kubernetes.io/projected/b4b0f202-374a-4e71-97db-50a6422c7767-kube-api-access-8mdlj\") pod \"ingress-canary-95h2r\" (UID: \"b4b0f202-374a-4e71-97db-50a6422c7767\") " pod="openshift-ingress-canary/ingress-canary-95h2r" Jul 14 19:11:21.333126 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.332945 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.333126 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.332969 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfkkm\" (UniqueName: \"kubernetes.io/projected/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-kube-api-access-kfkkm\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.333126 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.332994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e4919425-84e6-4f07-821c-87d4af4e71af-klusterlet-config\") pod \"klusterlet-addon-workmgr-789b67ddd8-v7lrh\" (UID: \"e4919425-84e6-4f07-821c-87d4af4e71af\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:11:21.333126 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:21.333005 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:11:21.333126 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:21.333022 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f44db5d46-9wzvr: secret "image-registry-tls" not found Jul 14 19:11:21.333126 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333023 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2c03a20-3487-4987-b514-d80fbb77da40-trusted-ca\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.333126 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333063 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/005d066e-4e27-42af-8e47-e6691b93f148-config-volume\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:21.333126 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333087 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:21.333126 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:21.333110 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls podName:b2c03a20-3487-4987-b514-d80fbb77da40 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:21.833088245 +0000 UTC m=+31.198158585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls") pod "image-registry-7f44db5d46-9wzvr" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40") : secret "image-registry-tls" not found Jul 14 19:11:21.333789 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:21.333167 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:11:21.333789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2c03a20-3487-4987-b514-d80fbb77da40-installation-pull-secrets\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.333789 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:21.333224 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls podName:005d066e-4e27-42af-8e47-e6691b93f148 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:21.833208988 +0000 UTC m=+31.198279342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls") pod "dns-default-4kwvb" (UID: "005d066e-4e27-42af-8e47-e6691b93f148") : secret "dns-default-metrics-tls" not found Jul 14 19:11:21.333789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333247 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4qqw\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-kube-api-access-r4qqw\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.333789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333267 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4919425-84e6-4f07-821c-87d4af4e71af-tmp\") pod \"klusterlet-addon-workmgr-789b67ddd8-v7lrh\" (UID: \"e4919425-84e6-4f07-821c-87d4af4e71af\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:11:21.333789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333281 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-bound-sa-token\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.333789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333322 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005d066e-4e27-42af-8e47-e6691b93f148-tmp-dir\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:21.333789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333354 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4kxq\" (UniqueName: \"kubernetes.io/projected/e4919425-84e6-4f07-821c-87d4af4e71af-kube-api-access-p4kxq\") pod \"klusterlet-addon-workmgr-789b67ddd8-v7lrh\" (UID: \"e4919425-84e6-4f07-821c-87d4af4e71af\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:11:21.333789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333405 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2c03a20-3487-4987-b514-d80fbb77da40-registry-certificates\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.333789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333433 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4b0f202-374a-4e71-97db-50a6422c7767-cert\") pod \"ingress-canary-95h2r\" (UID: \"b4b0f202-374a-4e71-97db-50a6422c7767\") " pod="openshift-ingress-canary/ingress-canary-95h2r" Jul 14 19:11:21.333789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333455 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-ca\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.333789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333478 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-hub\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.333789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333501 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.333789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.333568 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2c03a20-3487-4987-b514-d80fbb77da40-ca-trust-extracted\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.334637 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.334170 2572 generic.go:356] "Generic (PLEG): container finished" podID="6cedffc7-b943-486f-9322-73a3b01c7d93" containerID="85ea705ce68af2076f4c10cfa8c4efa5e9227462fc46f8e15b9baffc253dd0c4" exitCode=0 Jul 14 19:11:21.334637 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.334216 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/005d066e-4e27-42af-8e47-e6691b93f148-config-volume\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:21.334637 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.334369 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005d066e-4e27-42af-8e47-e6691b93f148-tmp-dir\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:21.334846 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.334825 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2c03a20-3487-4987-b514-d80fbb77da40-registry-certificates\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.334978 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.334954 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbjqz" event={"ID":"6cedffc7-b943-486f-9322-73a3b01c7d93","Type":"ContainerDied","Data":"85ea705ce68af2076f4c10cfa8c4efa5e9227462fc46f8e15b9baffc253dd0c4"} Jul 14 19:11:21.336787 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.336764 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.337003 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.336982 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2c03a20-3487-4987-b514-d80fbb77da40-installation-pull-secrets\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.337311 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.337230 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2c03a20-3487-4987-b514-d80fbb77da40-image-registry-private-configuration\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.337368 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.337319 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e4919425-84e6-4f07-821c-87d4af4e71af-klusterlet-config\") pod \"klusterlet-addon-workmgr-789b67ddd8-v7lrh\" (UID: \"e4919425-84e6-4f07-821c-87d4af4e71af\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:11:21.337599 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.337573 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.337788 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.337763 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-ca\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.337903 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.337854 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.338138 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.338118 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4b0f202-374a-4e71-97db-50a6422c7767-cert\") pod \"ingress-canary-95h2r\" (UID: \"b4b0f202-374a-4e71-97db-50a6422c7767\") " pod="openshift-ingress-canary/ingress-canary-95h2r" Jul 14 19:11:21.338727 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.338711 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-hub\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.340992 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.340969 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7stcj\" (UniqueName: \"kubernetes.io/projected/005d066e-4e27-42af-8e47-e6691b93f148-kube-api-access-7stcj\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:21.341120 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.341022 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2c03a20-3487-4987-b514-d80fbb77da40-trusted-ca\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.342155 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.342132 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfkkm\" (UniqueName: \"kubernetes.io/projected/fd0d0af9-9b9d-48db-98b3-b7a0bd65987d-kube-api-access-kfkkm\") pod \"cluster-proxy-proxy-agent-5dff65c897-qrf6g\" (UID: \"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.342251 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.342206 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-bound-sa-token\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.343322 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.343298 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4qqw\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-kube-api-access-r4qqw\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.344140 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.344111 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4kxq\" (UniqueName: \"kubernetes.io/projected/e4919425-84e6-4f07-821c-87d4af4e71af-kube-api-access-p4kxq\") pod \"klusterlet-addon-workmgr-789b67ddd8-v7lrh\" (UID: \"e4919425-84e6-4f07-821c-87d4af4e71af\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:11:21.344237 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.344219 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mdlj\" (UniqueName: \"kubernetes.io/projected/b4b0f202-374a-4e71-97db-50a6422c7767-kube-api-access-8mdlj\") pod \"ingress-canary-95h2r\" (UID: \"b4b0f202-374a-4e71-97db-50a6422c7767\") " pod="openshift-ingress-canary/ingress-canary-95h2r" Jul 14 19:11:21.380284 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.380248 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" Jul 14 19:11:21.387432 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.387400 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:11:21.431407 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.431377 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:11:21.457635 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.457604 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-95h2r" Jul 14 19:11:21.585486 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.585404 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j"] Jul 14 19:11:21.589987 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:11:21.589769 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3542f768_c658_451b_b911_b8956f9e8e39.slice/crio-f019f9ef4cbd03b3c42e219bdce07f9a12db98282d529bcefa00e9cb9d97632f WatchSource:0}: Error finding container f019f9ef4cbd03b3c42e219bdce07f9a12db98282d529bcefa00e9cb9d97632f: Status 404 returned error can't find the container with id f019f9ef4cbd03b3c42e219bdce07f9a12db98282d529bcefa00e9cb9d97632f Jul 14 19:11:21.591729 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.591701 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g"] Jul 14 19:11:21.594412 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.594389 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh"] Jul 14 19:11:21.596397 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:11:21.596373 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd0d0af9_9b9d_48db_98b3_b7a0bd65987d.slice/crio-69143d8ff0d065a9d6fe7ec30e807464c9262c2ddbf5e596a42958e69531412c WatchSource:0}: Error finding container 69143d8ff0d065a9d6fe7ec30e807464c9262c2ddbf5e596a42958e69531412c: Status 404 returned error can't find the container with id 69143d8ff0d065a9d6fe7ec30e807464c9262c2ddbf5e596a42958e69531412c Jul 14 19:11:21.597336 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:11:21.597309 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4919425_84e6_4f07_821c_87d4af4e71af.slice/crio-1acf11fe30b4103d5cc82f932e031028b2a181859acbaa0232d069ca8abe8d05 WatchSource:0}: Error finding container 1acf11fe30b4103d5cc82f932e031028b2a181859acbaa0232d069ca8abe8d05: Status 404 returned error can't find the container with id 1acf11fe30b4103d5cc82f932e031028b2a181859acbaa0232d069ca8abe8d05 Jul 14 19:11:21.619626 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.619596 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-95h2r"] Jul 14 19:11:21.622549 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:11:21.622524 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b0f202_374a_4e71_97db_50a6422c7767.slice/crio-70c3b2e438c6dbd749fc0cffa107c2614e772a7ea0c3036e6c398bc28c0261be WatchSource:0}: Error finding container 70c3b2e438c6dbd749fc0cffa107c2614e772a7ea0c3036e6c398bc28c0261be: Status 404 returned error can't find the container with id 70c3b2e438c6dbd749fc0cffa107c2614e772a7ea0c3036e6c398bc28c0261be Jul 14 19:11:21.839182 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.839091 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:21.839182 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:21.839138 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:21.839418 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:21.839245 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:11:21.839418 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:21.839264 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f44db5d46-9wzvr: secret "image-registry-tls" not found Jul 14 19:11:21.839418 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:21.839300 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:11:21.839418 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:21.839325 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls podName:b2c03a20-3487-4987-b514-d80fbb77da40 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:22.839309763 +0000 UTC m=+32.204380102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls") pod "image-registry-7f44db5d46-9wzvr" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40") : secret "image-registry-tls" not found Jul 14 19:11:21.839418 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:21.839349 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls podName:005d066e-4e27-42af-8e47-e6691b93f148 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:22.839334502 +0000 UTC m=+32.204404846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls") pod "dns-default-4kwvb" (UID: "005d066e-4e27-42af-8e47-e6691b93f148") : secret "dns-default-metrics-tls" not found Jul 14 19:11:22.341810 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:22.341764 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" event={"ID":"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d","Type":"ContainerStarted","Data":"69143d8ff0d065a9d6fe7ec30e807464c9262c2ddbf5e596a42958e69531412c"} Jul 14 19:11:22.344235 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:22.344153 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-95h2r" event={"ID":"b4b0f202-374a-4e71-97db-50a6422c7767","Type":"ContainerStarted","Data":"70c3b2e438c6dbd749fc0cffa107c2614e772a7ea0c3036e6c398bc28c0261be"} Jul 14 19:11:22.345979 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:22.345921 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" event={"ID":"3542f768-c658-451b-b911-b8956f9e8e39","Type":"ContainerStarted","Data":"f019f9ef4cbd03b3c42e219bdce07f9a12db98282d529bcefa00e9cb9d97632f"} Jul 14 19:11:22.347853 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:22.347797 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" event={"ID":"e4919425-84e6-4f07-821c-87d4af4e71af","Type":"ContainerStarted","Data":"1acf11fe30b4103d5cc82f932e031028b2a181859acbaa0232d069ca8abe8d05"} Jul 14 19:11:22.816131 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:22.815968 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-88fh2" Jul 14 19:11:22.816896 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:22.816391 2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 19:11:22.818275 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:22.817808 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-88fh2" Jul 14 19:11:22.849321 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:22.848471 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:22.849321 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:22.848600 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:22.849321 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:22.848729 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:11:22.849321 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:22.848743 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f44db5d46-9wzvr: secret "image-registry-tls" not found Jul 14 19:11:22.849321 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:22.848801 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls podName:b2c03a20-3487-4987-b514-d80fbb77da40 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:24.848781406 +0000 UTC m=+34.213851769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls") pod "image-registry-7f44db5d46-9wzvr" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40") : secret "image-registry-tls" not found Jul 14 19:11:22.849321 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:22.849208 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:11:22.849321 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:22.849257 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls podName:005d066e-4e27-42af-8e47-e6691b93f148 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:24.849241728 +0000 UTC m=+34.214312067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls") pod "dns-default-4kwvb" (UID: "005d066e-4e27-42af-8e47-e6691b93f148") : secret "dns-default-metrics-tls" not found Jul 14 19:11:22.883493 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:22.883369 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:24.768549 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:24.767338 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:24.768549 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:24.767486 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:24.768549 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:24.767614 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 14 19:11:24.768549 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:24.767676 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs podName:6e91d78e-4060-4063-86ac-5d7be516f269 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:56.767658716 +0000 UTC m=+66.132729060 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs") pod "network-metrics-daemon-h7v99" (UID: "6e91d78e-4060-4063-86ac-5d7be516f269") : secret "metrics-daemon-secret" not found Jul 14 19:11:24.775850 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:24.775175 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb3e121f-d03c-496e-8aad-591b078ec990-original-pull-secret\") pod \"global-pull-secret-syncer-t6nzh\" (UID: \"eb3e121f-d03c-496e-8aad-591b078ec990\") " pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:24.792845 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:24.792811 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6nzh" Jul 14 19:11:24.869070 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:24.868001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:24.869070 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:24.868091 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:24.869070 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:24.868123 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttkn\" (UniqueName: \"kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn\") pod \"network-check-target-h5x8l\" (UID: \"dfb34251-8e7e-4321-9a87-0dd7af1a459c\") " pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:24.869070 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:24.868767 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:11:24.869070 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:24.868786 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f44db5d46-9wzvr: secret "image-registry-tls" not found Jul 14 19:11:24.869070 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:24.868848 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls podName:b2c03a20-3487-4987-b514-d80fbb77da40 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:28.868827721 +0000 UTC m=+38.233898083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls") pod "image-registry-7f44db5d46-9wzvr" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40") : secret "image-registry-tls" not found Jul 14 19:11:24.869932 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:24.869585 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:11:24.869932 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:24.869666 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls podName:005d066e-4e27-42af-8e47-e6691b93f148 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:28.869648846 +0000 UTC m=+38.234719188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls") pod "dns-default-4kwvb" (UID: "005d066e-4e27-42af-8e47-e6691b93f148") : secret "dns-default-metrics-tls" not found Jul 14 19:11:24.875209 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:24.875142 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jttkn\" (UniqueName: \"kubernetes.io/projected/dfb34251-8e7e-4321-9a87-0dd7af1a459c-kube-api-access-jttkn\") pod \"network-check-target-h5x8l\" (UID: \"dfb34251-8e7e-4321-9a87-0dd7af1a459c\") " pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:25.099013 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:25.098707 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:25.831362 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:25.831226 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:25.929023 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:25.928984 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:26.029012 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:26.028961 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:26.946677 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:26.946641 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:27.047083 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:27.047027 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:27.586338 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:27.586301 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:27.687948 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:27.687911 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:28.899522 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:28.899486 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:28.900113 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:28.899530 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:28.900113 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:28.899647 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:11:28.900113 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:28.899658 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:11:28.900113 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:28.899682 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f44db5d46-9wzvr: secret "image-registry-tls" not found Jul 14 19:11:28.900113 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:28.899708 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls podName:005d066e-4e27-42af-8e47-e6691b93f148 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:36.899694566 +0000 UTC m=+46.264764905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls") pod "dns-default-4kwvb" (UID: "005d066e-4e27-42af-8e47-e6691b93f148") : secret "dns-default-metrics-tls" not found Jul 14 19:11:28.900113 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:28.899738 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls podName:b2c03a20-3487-4987-b514-d80fbb77da40 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:36.899719898 +0000 UTC m=+46.264790236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls") pod "image-registry-7f44db5d46-9wzvr" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40") : secret "image-registry-tls" not found Jul 14 19:11:29.294882 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:29.294805 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:29.369692 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:29.369646 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:31.585588 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:31.585561 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t6nzh"] Jul 14 19:11:31.603079 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:31.603006 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h5x8l"] Jul 14 19:11:31.718634 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:11:31.718546 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3e121f_d03c_496e_8aad_591b078ec990.slice/crio-64bb68b749e6ea102a9935347808b6c9146fa2a8689624ebecd26b0041bb08e6 WatchSource:0}: Error finding container 64bb68b749e6ea102a9935347808b6c9146fa2a8689624ebecd26b0041bb08e6: Status 404 returned error can't find the container with id 64bb68b749e6ea102a9935347808b6c9146fa2a8689624ebecd26b0041bb08e6 Jul 14 19:11:31.719204 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:11:31.719177 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb34251_8e7e_4321_9a87_0dd7af1a459c.slice/crio-4894329b3bb5f69b405e94662c6f6ffb2ff6cb7bfb6d255a719d24bfc3bc4f13 WatchSource:0}: Error finding container 4894329b3bb5f69b405e94662c6f6ffb2ff6cb7bfb6d255a719d24bfc3bc4f13: Status 404 returned error can't find the container with id 4894329b3bb5f69b405e94662c6f6ffb2ff6cb7bfb6d255a719d24bfc3bc4f13 Jul 14 19:11:32.380022 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.379978 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t6nzh" event={"ID":"eb3e121f-d03c-496e-8aad-591b078ec990","Type":"ContainerStarted","Data":"64bb68b749e6ea102a9935347808b6c9146fa2a8689624ebecd26b0041bb08e6"} Jul 14 19:11:32.382304 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.382267 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" event={"ID":"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d","Type":"ContainerStarted","Data":"a53b92cba8ca269e6cd9cbac7c90d5ac13c2e9c7e40bd8b32931b8506a933777"} Jul 14 19:11:32.382474 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.382315 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" event={"ID":"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d","Type":"ContainerStarted","Data":"a7297f0a8c69c51b14485ccb75d70c0f022aaadbf8e825f9d48f8292cd675d35"} Jul 14 19:11:32.382474 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.382332 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" event={"ID":"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d","Type":"ContainerStarted","Data":"54811bfd4cf818bad6c22843b99c6dc7c6416d8f831e9cd26c8429bbfab9e5af"} Jul 14 19:11:32.383750 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.383720 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h5x8l" event={"ID":"dfb34251-8e7e-4321-9a87-0dd7af1a459c","Type":"ContainerStarted","Data":"4894329b3bb5f69b405e94662c6f6ffb2ff6cb7bfb6d255a719d24bfc3bc4f13"} Jul 14 19:11:32.385331 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.385300 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-95h2r" event={"ID":"b4b0f202-374a-4e71-97db-50a6422c7767","Type":"ContainerStarted","Data":"3823ac9298d0a6a6462ed74cb540a90721b2cc88d775707309c08431fc3fd1c4"} Jul 14 19:11:32.386890 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.386866 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" event={"ID":"3542f768-c658-451b-b911-b8956f9e8e39","Type":"ContainerStarted","Data":"2d90a128defd2dc0b45dbf7c5d16e5bfe94814e02ce378c1e88841f7e9cde653"} Jul 14 19:11:32.389849 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.389824 2572 generic.go:356] "Generic (PLEG): container finished" podID="6cedffc7-b943-486f-9322-73a3b01c7d93" containerID="ae4e8a1daceaa3569c2c6f9071d36785d0c8abc151d3abb27d1864abdfb0627a" exitCode=0 Jul 14 19:11:32.389967 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.389912 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbjqz" event={"ID":"6cedffc7-b943-486f-9322-73a3b01c7d93","Type":"ContainerDied","Data":"ae4e8a1daceaa3569c2c6f9071d36785d0c8abc151d3abb27d1864abdfb0627a"} Jul 14 19:11:32.391405 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.391382 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" event={"ID":"e4919425-84e6-4f07-821c-87d4af4e71af","Type":"ContainerStarted","Data":"5a3b6f49a6401aed2f0415a7d8d0536753e10c04695876a73efb49186aa11ae3"} Jul 14 19:11:32.391598 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.391582 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:11:32.393645 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.393628 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:11:32.403583 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.403539 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" podStartSLOduration=7.543517979 podStartE2EDuration="17.403521419s" totalImagesPullingTime="9.86000344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:32.403222962 +0000 UTC m=+41.768293315" watchObservedRunningTime="2026-07-14 19:11:32.403521419 +0000 UTC m=+41.768591779" Jul 14 19:11:32.446241 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.445985 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-95h2r" podStartSLOduration=1.6311316169999999 podStartE2EDuration="11.445964694s" totalImagesPullingTime="9.814833077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:32.4224313 +0000 UTC m=+41.787501661" watchObservedRunningTime="2026-07-14 19:11:32.445964694 +0000 UTC m=+41.811035055" Jul 14 19:11:32.461792 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.461734 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" podStartSLOduration=7.615913685 podStartE2EDuration="17.461711949s" totalImagesPullingTime="9.845798264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:32.460406148 +0000 UTC m=+41.825476512" watchObservedRunningTime="2026-07-14 19:11:32.461711949 +0000 UTC m=+41.826782306" Jul 14 19:11:32.477579 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:32.476864 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" podStartSLOduration=7.637101505 podStartE2EDuration="17.476841723s" totalImagesPullingTime="9.839740218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:32.476450815 +0000 UTC m=+41.841521200" watchObservedRunningTime="2026-07-14 19:11:32.476841723 +0000 UTC m=+41.841912085" Jul 14 19:11:33.399168 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:33.399129 2572 generic.go:356] "Generic (PLEG): container finished" podID="6cedffc7-b943-486f-9322-73a3b01c7d93" containerID="86dae0dd1ba1b75f59a0794472acb3d9afabc3e5268bc4f1db1e3feb950ebed0" exitCode=0 Jul 14 19:11:33.400243 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:33.399267 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbjqz" event={"ID":"6cedffc7-b943-486f-9322-73a3b01c7d93","Type":"ContainerDied","Data":"86dae0dd1ba1b75f59a0794472acb3d9afabc3e5268bc4f1db1e3feb950ebed0"} Jul 14 19:11:34.407074 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:34.406979 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbjqz" event={"ID":"6cedffc7-b943-486f-9322-73a3b01c7d93","Type":"ContainerStarted","Data":"d950b2494d06b97db42a1358d9906ff6b946193006b7154e360e0a6d5b95c369"} Jul 14 19:11:34.431694 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:34.431633 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tbjqz" podStartSLOduration=10.121329631 podStartE2EDuration="43.431610028s" totalImagesPullingTime="33.310280397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:34.430760781 +0000 UTC m=+43.795831143" watchObservedRunningTime="2026-07-14 19:11:34.431610028 +0000 UTC m=+43.796680391" Jul 14 19:11:34.552547 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:34.552511 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:34.653758 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:34.653663 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:36.978990 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:36.978946 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:36.978990 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:36.978993 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:36.979475 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:36.979108 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:11:36.979475 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:36.979134 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f44db5d46-9wzvr: secret "image-registry-tls" not found Jul 14 19:11:36.979475 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:36.979185 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls podName:b2c03a20-3487-4987-b514-d80fbb77da40 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:52.979169053 +0000 UTC m=+62.344239392 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls") pod "image-registry-7f44db5d46-9wzvr" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40") : secret "image-registry-tls" not found Jul 14 19:11:36.979475 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:36.979116 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:11:36.979475 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:36.979242 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls podName:005d066e-4e27-42af-8e47-e6691b93f148 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:52.979229372 +0000 UTC m=+62.344299727 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls") pod "dns-default-4kwvb" (UID: "005d066e-4e27-42af-8e47-e6691b93f148") : secret "dns-default-metrics-tls" not found Jul 14 19:11:37.419350 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:37.419246 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h5x8l" event={"ID":"dfb34251-8e7e-4321-9a87-0dd7af1a459c","Type":"ContainerStarted","Data":"8bf128e61059e14a9f2fff254fad60733bab31ce164f4416e9add6f17d355f6b"} Jul 14 19:11:37.419518 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:37.419408 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:11:37.420762 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:37.420738 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t6nzh" event={"ID":"eb3e121f-d03c-496e-8aad-591b078ec990","Type":"ContainerStarted","Data":"ee08a5b484fa2d7003949394a4c6266cd02f77e5b74929442f75a43df94e4c3a"} Jul 14 19:11:37.430343 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:37.430319 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:37.435345 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:37.435305 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h5x8l" podStartSLOduration=41.375054536 podStartE2EDuration="46.435283526s" totalImagesPullingTime="5.06022899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:37.434424236 +0000 UTC m=+46.799494598" watchObservedRunningTime="2026-07-14 19:11:37.435283526 +0000 UTC m=+46.800353887" Jul 14 19:11:37.450094 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:37.450030 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t6nzh" podStartSLOduration=41.045399162 podStartE2EDuration="46.450014805s" totalImagesPullingTime="5.404615643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:37.448985043 +0000 UTC m=+46.814055416" watchObservedRunningTime="2026-07-14 19:11:37.450014805 +0000 UTC m=+46.815085166" Jul 14 19:11:37.532577 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:37.532539 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:49.827909 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:49.827868 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:49.926614 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:49.926575 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:52.989248 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:52.989197 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:11:52.989248 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:52.989251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:11:52.989758 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:52.989368 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:11:52.989758 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:52.989373 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:11:52.989758 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:52.989392 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f44db5d46-9wzvr: secret "image-registry-tls" not found Jul 14 19:11:52.989758 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:52.989440 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls podName:005d066e-4e27-42af-8e47-e6691b93f148 nodeName:}" failed. No retries permitted until 2026-07-14 19:12:24.989420655 +0000 UTC m=+94.354490995 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls") pod "dns-default-4kwvb" (UID: "005d066e-4e27-42af-8e47-e6691b93f148") : secret "dns-default-metrics-tls" not found Jul 14 19:11:52.989758 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:52.989453 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls podName:b2c03a20-3487-4987-b514-d80fbb77da40 nodeName:}" failed. No retries permitted until 2026-07-14 19:12:24.989447262 +0000 UTC m=+94.354517600 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls") pod "image-registry-7f44db5d46-9wzvr" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40") : secret "image-registry-tls" not found Jul 14 19:11:56.817165 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:56.817117 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:11:56.817571 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:56.817275 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 14 19:11:56.817571 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:11:56.817343 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs podName:6e91d78e-4060-4063-86ac-5d7be516f269 nodeName:}" failed. No retries permitted until 2026-07-14 19:13:00.81732601 +0000 UTC m=+130.182396349 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs") pod "network-metrics-daemon-h7v99" (UID: "6e91d78e-4060-4063-86ac-5d7be516f269") : secret "metrics-daemon-secret" not found Jul 14 19:11:57.234422 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:57.234388 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:57.296361 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:57.296328 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:11:57.355162 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:11:57.355125 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:12:01.243306 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:12:01.243272 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:12:01.326920 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:12:01.326885 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:12:08.426438 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:12:08.426407 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h5x8l" Jul 14 19:12:25.020179 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:12:25.020136 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:12:25.020179 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:12:25.020183 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:12:25.020606 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:12:25.020281 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:12:25.020606 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:12:25.020286 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:12:25.020606 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:12:25.020308 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f44db5d46-9wzvr: secret "image-registry-tls" not found Jul 14 19:12:25.020606 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:12:25.020345 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls podName:005d066e-4e27-42af-8e47-e6691b93f148 nodeName:}" failed. No retries permitted until 2026-07-14 19:13:29.020327875 +0000 UTC m=+158.385398214 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls") pod "dns-default-4kwvb" (UID: "005d066e-4e27-42af-8e47-e6691b93f148") : secret "dns-default-metrics-tls" not found Jul 14 19:12:25.020606 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:12:25.020359 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls podName:b2c03a20-3487-4987-b514-d80fbb77da40 nodeName:}" failed. No retries permitted until 2026-07-14 19:13:29.020352883 +0000 UTC m=+158.385423222 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls") pod "image-registry-7f44db5d46-9wzvr" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40") : secret "image-registry-tls" not found Jul 14 19:12:27.633716 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:12:27.633680 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:12:27.720659 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:12:27.720618 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:13:00.891929 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:00.891876 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:13:00.892486 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:13:00.892025 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 14 19:13:00.892486 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:13:00.892112 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs podName:6e91d78e-4060-4063-86ac-5d7be516f269 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:02.89209509 +0000 UTC m=+252.257165432 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs") pod "network-metrics-daemon-h7v99" (UID: "6e91d78e-4060-4063-86ac-5d7be516f269") : secret "metrics-daemon-secret" not found Jul 14 19:13:02.570635 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:02.570595 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:13:24.124844 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:13:24.124801 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" podUID="b2c03a20-3487-4987-b514-d80fbb77da40" Jul 14 19:13:24.148298 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:13:24.148251 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4kwvb" podUID="005d066e-4e27-42af-8e47-e6691b93f148" Jul 14 19:13:24.203979 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:13:24.203945 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:13:24.718158 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:24.718127 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:13:24.718320 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:24.718127 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4kwvb" Jul 14 19:13:29.106964 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:29.106929 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:13:29.106964 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:29.106967 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:13:29.107475 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:13:29.107093 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:13:29.107475 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:13:29.107129 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:13:29.107475 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:13:29.107153 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls podName:005d066e-4e27-42af-8e47-e6691b93f148 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:31.107138403 +0000 UTC m=+280.472208743 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls") pod "dns-default-4kwvb" (UID: "005d066e-4e27-42af-8e47-e6691b93f148") : secret "dns-default-metrics-tls" not found Jul 14 19:13:29.107475 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:13:29.107154 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f44db5d46-9wzvr: secret "image-registry-tls" not found Jul 14 19:13:29.107475 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:13:29.107226 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls podName:b2c03a20-3487-4987-b514-d80fbb77da40 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:31.107207163 +0000 UTC m=+280.472277501 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls") pod "image-registry-7f44db5d46-9wzvr" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40") : secret "image-registry-tls" not found Jul 14 19:13:31.739019 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:31.738988 2572 generic.go:356] "Generic (PLEG): container finished" podID="e4919425-84e6-4f07-821c-87d4af4e71af" containerID="5a3b6f49a6401aed2f0415a7d8d0536753e10c04695876a73efb49186aa11ae3" exitCode=1 Jul 14 19:13:31.739471 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:31.739067 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" event={"ID":"e4919425-84e6-4f07-821c-87d4af4e71af","Type":"ContainerDied","Data":"5a3b6f49a6401aed2f0415a7d8d0536753e10c04695876a73efb49186aa11ae3"} Jul 14 19:13:31.739471 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:31.739464 2572 scope.go:117] "RemoveContainer" containerID="5a3b6f49a6401aed2f0415a7d8d0536753e10c04695876a73efb49186aa11ae3" Jul 14 19:13:31.740723 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:31.740593 2572 generic.go:356] "Generic (PLEG): container finished" podID="3542f768-c658-451b-b911-b8956f9e8e39" containerID="2d90a128defd2dc0b45dbf7c5d16e5bfe94814e02ce378c1e88841f7e9cde653" exitCode=255 Jul 14 19:13:31.740723 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:31.740628 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" event={"ID":"3542f768-c658-451b-b911-b8956f9e8e39","Type":"ContainerDied","Data":"2d90a128defd2dc0b45dbf7c5d16e5bfe94814e02ce378c1e88841f7e9cde653"} Jul 14 19:13:31.740848 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:31.740837 2572 scope.go:117] "RemoveContainer" containerID="2d90a128defd2dc0b45dbf7c5d16e5bfe94814e02ce378c1e88841f7e9cde653" Jul 14 19:13:32.392595 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:32.392559 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:13:32.745724 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:32.745629 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" event={"ID":"e4919425-84e6-4f07-821c-87d4af4e71af","Type":"ContainerStarted","Data":"32cb2f49cf364963c2d79a198a2a122eefc363a22a1e1ac2b381b818ee3a99d3"} Jul 14 19:13:32.746170 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:32.745799 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:13:32.746565 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:32.746548 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:13:32.747369 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:32.747350 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" event={"ID":"3542f768-c658-451b-b911-b8956f9e8e39","Type":"ContainerStarted","Data":"ef322abd1128b799484dd5f7c4eb939cae34f7bceb39132fe850db1fba99ef95"} Jul 14 19:13:36.183244 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:13:36.183196 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:14:18.799231 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:18.799148 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:14:28.689897 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:28.689864 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:14:51.765109 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:51.765034 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:14:52.382400 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:52.382363 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:14:53.745901 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:53.745861 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:14:53.847680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:53.847651 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:14:56.399994 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.399963 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:14:56.473632 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.473608 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8zltv"] Jul 14 19:14:56.475495 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.475479 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:56.484187 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.484168 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 19:14:56.490207 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.490189 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 19:14:56.490313 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.490214 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 19:14:56.490313 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.490228 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9hbmm\"" Jul 14 19:14:56.492642 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.492626 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 19:14:56.500643 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.500623 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8zltv"] Jul 14 19:14:56.517830 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.517803 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:14:56.534986 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.534964 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:56.535096 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.535007 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/645c204e-e9c0-4963-a53a-4192c14caaae-data-volume\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:56.535096 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.535031 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:56.535215 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.535140 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvsc6\" (UniqueName: \"kubernetes.io/projected/645c204e-e9c0-4963-a53a-4192c14caaae-kube-api-access-xvsc6\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:56.535215 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.535193 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/645c204e-e9c0-4963-a53a-4192c14caaae-crio-socket\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:56.605433 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.605399 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:14:56.635927 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.635903 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/645c204e-e9c0-4963-a53a-4192c14caaae-data-volume\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:56.636015 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.635941 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:56.636015 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.635997 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvsc6\" (UniqueName: \"kubernetes.io/projected/645c204e-e9c0-4963-a53a-4192c14caaae-kube-api-access-xvsc6\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:56.636147 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.636037 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/645c204e-e9c0-4963-a53a-4192c14caaae-crio-socket\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:56.636147 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.636098 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:56.636252 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:14:56.636153 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:14:56.636252 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.636177 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/645c204e-e9c0-4963-a53a-4192c14caaae-crio-socket\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:56.636252 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:14:56.636198 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 19:14:56.636252 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:14:56.636232 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls podName:645c204e-e9c0-4963-a53a-4192c14caaae nodeName:}" failed. No retries permitted until 2026-07-14 19:14:57.136211952 +0000 UTC m=+246.501282300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zltv" (UID: "645c204e-e9c0-4963-a53a-4192c14caaae") : secret "insights-runtime-extractor-tls" not found Jul 14 19:14:56.636449 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:14:56.636262 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm podName:645c204e-e9c0-4963-a53a-4192c14caaae nodeName:}" failed. No retries permitted until 2026-07-14 19:14:57.136245591 +0000 UTC m=+246.501315931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zltv" (UID: "645c204e-e9c0-4963-a53a-4192c14caaae") : configmap "kube-rbac-proxy" not found Jul 14 19:14:56.636449 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.636335 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/645c204e-e9c0-4963-a53a-4192c14caaae-data-volume\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:56.645534 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.645512 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvsc6\" (UniqueName: \"kubernetes.io/projected/645c204e-e9c0-4963-a53a-4192c14caaae-kube-api-access-xvsc6\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:56.705770 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:56.705707 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:14:57.074275 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:57.074185 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:14:57.140188 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:57.140158 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:57.140349 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:57.140223 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:57.140349 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:14:57.140307 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 19:14:57.140349 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:14:57.140318 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:14:57.140502 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:14:57.140361 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm podName:645c204e-e9c0-4963-a53a-4192c14caaae nodeName:}" failed. No retries permitted until 2026-07-14 19:14:58.140344095 +0000 UTC m=+247.505414434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zltv" (UID: "645c204e-e9c0-4963-a53a-4192c14caaae") : configmap "kube-rbac-proxy" not found Jul 14 19:14:57.140502 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:14:57.140375 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls podName:645c204e-e9c0-4963-a53a-4192c14caaae nodeName:}" failed. No retries permitted until 2026-07-14 19:14:58.140368497 +0000 UTC m=+247.505438835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zltv" (UID: "645c204e-e9c0-4963-a53a-4192c14caaae") : secret "insights-runtime-extractor-tls" not found Jul 14 19:14:57.172691 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:57.172662 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:14:58.148211 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:58.148173 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:58.148632 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:58.148264 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:14:58.148632 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:14:58.148336 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:14:58.148739 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:14:58.148698 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 19:14:58.148790 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:14:58.148740 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls podName:645c204e-e9c0-4963-a53a-4192c14caaae nodeName:}" failed. No retries permitted until 2026-07-14 19:15:00.148710928 +0000 UTC m=+249.513781288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zltv" (UID: "645c204e-e9c0-4963-a53a-4192c14caaae") : secret "insights-runtime-extractor-tls" not found Jul 14 19:14:58.148790 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:14:58.148779 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm podName:645c204e-e9c0-4963-a53a-4192c14caaae nodeName:}" failed. No retries permitted until 2026-07-14 19:15:00.148765509 +0000 UTC m=+249.513835854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zltv" (UID: "645c204e-e9c0-4963-a53a-4192c14caaae") : configmap "kube-rbac-proxy" not found Jul 14 19:14:58.629362 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:58.629339 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vq5gr_8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f/dns-node-resolver/0.log" Jul 14 19:14:59.230004 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:14:59.229970 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gndd8_a6bb4d98-4bbc-4c23-82ad-351fdd921751/node-ca/0.log" Jul 14 19:15:00.028909 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:00.028883 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-95h2r_b4b0f202-374a-4e71-97db-50a6422c7767/serve-healthcheck-canary/0.log" Jul 14 19:15:00.161794 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:00.161754 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:15:00.161981 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:00.161827 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:15:00.161981 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:00.161879 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 19:15:00.161981 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:00.161951 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm podName:645c204e-e9c0-4963-a53a-4192c14caaae nodeName:}" failed. No retries permitted until 2026-07-14 19:15:04.161934624 +0000 UTC m=+253.527004962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zltv" (UID: "645c204e-e9c0-4963-a53a-4192c14caaae") : configmap "kube-rbac-proxy" not found Jul 14 19:15:00.161981 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:00.161971 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:15:00.162203 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:00.162085 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls podName:645c204e-e9c0-4963-a53a-4192c14caaae nodeName:}" failed. No retries permitted until 2026-07-14 19:15:04.162067607 +0000 UTC m=+253.527137947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zltv" (UID: "645c204e-e9c0-4963-a53a-4192c14caaae") : secret "insights-runtime-extractor-tls" not found Jul 14 19:15:01.128567 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:01.128532 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:01.242718 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:01.242682 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:02.118884 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:02.118847 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:02.605354 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:02.605319 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:02.746388 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:02.746353 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:02.940338 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:02.940253 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:02.985172 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:02.985137 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:15:02.985325 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:02.985288 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 14 19:15:02.985375 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:02.985352 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs podName:6e91d78e-4060-4063-86ac-5d7be516f269 nodeName:}" failed. No retries permitted until 2026-07-14 19:17:04.985336497 +0000 UTC m=+374.350406837 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs") pod "network-metrics-daemon-h7v99" (UID: "6e91d78e-4060-4063-86ac-5d7be516f269") : secret "metrics-daemon-secret" not found Jul 14 19:15:03.345488 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:03.345448 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:03.619163 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:03.619080 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:04.193542 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:04.193507 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:15:04.193683 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:04.193561 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:15:04.193683 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:04.193630 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 19:15:04.193683 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:04.193678 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:15:04.193803 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:04.193688 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm podName:645c204e-e9c0-4963-a53a-4192c14caaae nodeName:}" failed. No retries permitted until 2026-07-14 19:15:12.193674341 +0000 UTC m=+261.558744679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zltv" (UID: "645c204e-e9c0-4963-a53a-4192c14caaae") : configmap "kube-rbac-proxy" not found Jul 14 19:15:04.193803 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:04.193728 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls podName:645c204e-e9c0-4963-a53a-4192c14caaae nodeName:}" failed. No retries permitted until 2026-07-14 19:15:12.193717332 +0000 UTC m=+261.558787671 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zltv" (UID: "645c204e-e9c0-4963-a53a-4192c14caaae") : secret "insights-runtime-extractor-tls" not found Jul 14 19:15:06.388819 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:06.388771 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:06.461843 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:06.461797 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:06.559845 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:06.559813 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:06.912670 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:06.912628 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:07.008583 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:07.008551 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:07.825696 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:07.825659 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:08.138466 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:08.138377 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:08.336812 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:08.336780 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:08.794351 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:08.794322 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:11.634228 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:11.634193 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:11.736620 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:11.736591 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:12.263310 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:12.263205 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:15:12.263310 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:12.263279 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:15:12.263832 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:12.263806 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/645c204e-e9c0-4963-a53a-4192c14caaae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:15:12.265525 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:12.265508 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/645c204e-e9c0-4963-a53a-4192c14caaae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zltv\" (UID: \"645c204e-e9c0-4963-a53a-4192c14caaae\") " pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:15:12.360426 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:12.360399 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:12.383420 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:12.383388 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8zltv" Jul 14 19:15:12.501695 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:12.501666 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8zltv"] Jul 14 19:15:12.504569 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:15:12.504545 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645c204e_e9c0_4963_a53a_4192c14caaae.slice/crio-f89f852386fe9619c7e086dc2fb059d0b95df04c0410849f72e86cdd33e88b20 WatchSource:0}: Error finding container f89f852386fe9619c7e086dc2fb059d0b95df04c0410849f72e86cdd33e88b20: Status 404 returned error can't find the container with id f89f852386fe9619c7e086dc2fb059d0b95df04c0410849f72e86cdd33e88b20 Jul 14 19:15:13.031734 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:13.031653 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zltv" event={"ID":"645c204e-e9c0-4963-a53a-4192c14caaae","Type":"ContainerStarted","Data":"f89f852386fe9619c7e086dc2fb059d0b95df04c0410849f72e86cdd33e88b20"} Jul 14 19:15:16.538617 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:16.538524 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:16.639901 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:16.639866 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:16.846691 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:16.846655 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:17.045357 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:17.045263 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zltv" event={"ID":"645c204e-e9c0-4963-a53a-4192c14caaae","Type":"ContainerStarted","Data":"d57d8818f8fae348bf53c68717835450b3a0c2163a7abcbcb58bc01f8cd3c966"} Jul 14 19:15:17.045357 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:17.045325 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zltv" event={"ID":"645c204e-e9c0-4963-a53a-4192c14caaae","Type":"ContainerStarted","Data":"c32f6e0aa1406f8780e038bd53b208b2f0b24272f51c079ca2b938ecf726bec4"} Jul 14 19:15:17.419129 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:17.419084 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:17.491092 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:17.491059 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:17.746807 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:17.746702 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:17.857476 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:17.857423 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:18.228576 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:18.228532 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:18.424240 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:18.424198 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:19.010013 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:19.009980 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:19.466371 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:19.466331 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:19.564276 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:19.564237 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:20.056298 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:20.056266 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zltv" event={"ID":"645c204e-e9c0-4963-a53a-4192c14caaae","Type":"ContainerStarted","Data":"f552bd4c738318a75b2aee6bfe8b9a2ce9a0b2989e8f6bbd8810ab11c30169a4"} Jul 14 19:15:20.075379 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:20.075324 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8zltv" podStartSLOduration=17.11328525 podStartE2EDuration="24.075310153s" totalImagesPullingTime="6.962024903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:14:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:20.074332743 +0000 UTC m=+269.439403108" watchObservedRunningTime="2026-07-14 19:15:20.075310153 +0000 UTC m=+269.440380513" Jul 14 19:15:21.388426 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:21.388388 2572 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" podUID="fd0d0af9-9b9d-48db-98b3-b7a0bd65987d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 14 19:15:25.416657 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:25.416615 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:25.493966 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:25.493931 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:25.644879 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:25.644842 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:25.879712 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:25.879678 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:25.982424 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:25.982391 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:27.719110 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:27.719066 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" podUID="b2c03a20-3487-4987-b514-d80fbb77da40" Jul 14 19:15:27.719110 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:27.719072 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4kwvb" podUID="005d066e-4e27-42af-8e47-e6691b93f148" Jul 14 19:15:28.080163 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:28.080136 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4kwvb" Jul 14 19:15:28.080316 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:28.080136 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:15:29.661969 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:29.661925 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:29.759957 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:29.759917 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:31.108276 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:31.108243 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:15:31.108276 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:31.108278 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:15:31.110685 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:31.110662 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/005d066e-4e27-42af-8e47-e6691b93f148-metrics-tls\") pod \"dns-default-4kwvb\" (UID: \"005d066e-4e27-42af-8e47-e6691b93f148\") " pod="openshift-dns/dns-default-4kwvb" Jul 14 19:15:31.110753 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:31.110739 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls\") pod \"image-registry-7f44db5d46-9wzvr\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:15:31.384756 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:31.384679 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5x4l\"" Jul 14 19:15:31.384756 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:31.384682 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-plsql\"" Jul 14 19:15:31.388432 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:31.388395 2572 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" podUID="fd0d0af9-9b9d-48db-98b3-b7a0bd65987d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 14 19:15:31.391481 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:31.391452 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4kwvb" Jul 14 19:15:31.391591 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:31.391464 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:15:31.537192 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:31.537162 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f44db5d46-9wzvr"] Jul 14 19:15:31.540861 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:15:31.540829 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c03a20_3487_4987_b514_d80fbb77da40.slice/crio-1c2a9514ac60e222122593b50e3d1a656c0f28e6beb60d44744a9d7d980b53ce WatchSource:0}: Error finding container 1c2a9514ac60e222122593b50e3d1a656c0f28e6beb60d44744a9d7d980b53ce: Status 404 returned error can't find the container with id 1c2a9514ac60e222122593b50e3d1a656c0f28e6beb60d44744a9d7d980b53ce Jul 14 19:15:31.550550 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:31.550528 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4kwvb"] Jul 14 19:15:31.553138 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:15:31.553113 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005d066e_4e27_42af_8e47_e6691b93f148.slice/crio-1758c837afa29c506cb52f4ea25d30f1f593b0e55a9cff278bdc51df53547eae WatchSource:0}: Error finding container 1758c837afa29c506cb52f4ea25d30f1f593b0e55a9cff278bdc51df53547eae: Status 404 returned error can't find the container with id 1758c837afa29c506cb52f4ea25d30f1f593b0e55a9cff278bdc51df53547eae Jul 14 19:15:31.867071 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:31.867024 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:31.946514 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:31.946474 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:32.094140 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:32.094060 2572 generic.go:356] "Generic (PLEG): container finished" podID="3542f768-c658-451b-b911-b8956f9e8e39" containerID="ef322abd1128b799484dd5f7c4eb939cae34f7bceb39132fe850db1fba99ef95" exitCode=255 Jul 14 19:15:32.094140 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:32.094119 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" event={"ID":"3542f768-c658-451b-b911-b8956f9e8e39","Type":"ContainerDied","Data":"ef322abd1128b799484dd5f7c4eb939cae34f7bceb39132fe850db1fba99ef95"} Jul 14 19:15:32.094938 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:32.094921 2572 scope.go:117] "RemoveContainer" containerID="2d90a128defd2dc0b45dbf7c5d16e5bfe94814e02ce378c1e88841f7e9cde653" Jul 14 19:15:32.095055 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:32.094999 2572 scope.go:117] "RemoveContainer" containerID="ef322abd1128b799484dd5f7c4eb939cae34f7bceb39132fe850db1fba99ef95" Jul 14 19:15:32.095219 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:32.095196 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j_open-cluster-management-agent-addon(3542f768-c658-451b-b911-b8956f9e8e39)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" podUID="3542f768-c658-451b-b911-b8956f9e8e39" Jul 14 19:15:32.095705 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:32.095684 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" event={"ID":"b2c03a20-3487-4987-b514-d80fbb77da40","Type":"ContainerStarted","Data":"f95f1eaaa2b9143b8efe07079976b267c5c88fbbb0a8569d5e33b9d26e7bf842"} Jul 14 19:15:32.095814 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:32.095725 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:15:32.095814 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:32.095739 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" event={"ID":"b2c03a20-3487-4987-b514-d80fbb77da40","Type":"ContainerStarted","Data":"1c2a9514ac60e222122593b50e3d1a656c0f28e6beb60d44744a9d7d980b53ce"} Jul 14 19:15:32.097334 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:32.097290 2572 generic.go:356] "Generic (PLEG): container finished" podID="e4919425-84e6-4f07-821c-87d4af4e71af" containerID="32cb2f49cf364963c2d79a198a2a122eefc363a22a1e1ac2b381b818ee3a99d3" exitCode=1 Jul 14 19:15:32.097444 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:32.097362 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" event={"ID":"e4919425-84e6-4f07-821c-87d4af4e71af","Type":"ContainerDied","Data":"32cb2f49cf364963c2d79a198a2a122eefc363a22a1e1ac2b381b818ee3a99d3"} Jul 14 19:15:32.097705 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:32.097685 2572 scope.go:117] "RemoveContainer" containerID="32cb2f49cf364963c2d79a198a2a122eefc363a22a1e1ac2b381b818ee3a99d3" Jul 14 19:15:32.097856 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:32.097838 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-789b67ddd8-v7lrh_open-cluster-management-agent-addon(e4919425-84e6-4f07-821c-87d4af4e71af)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" podUID="e4919425-84e6-4f07-821c-87d4af4e71af" Jul 14 19:15:32.098389 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:32.098372 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4kwvb" event={"ID":"005d066e-4e27-42af-8e47-e6691b93f148","Type":"ContainerStarted","Data":"1758c837afa29c506cb52f4ea25d30f1f593b0e55a9cff278bdc51df53547eae"} Jul 14 19:15:32.106355 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:32.106342 2572 scope.go:117] "RemoveContainer" containerID="5a3b6f49a6401aed2f0415a7d8d0536753e10c04695876a73efb49186aa11ae3" Jul 14 19:15:32.147438 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:32.147387 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" podStartSLOduration=284.147374692 podStartE2EDuration="4m44.147374692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:32.147091807 +0000 UTC m=+281.512162170" watchObservedRunningTime="2026-07-14 19:15:32.147374692 +0000 UTC m=+281.512445052" Jul 14 19:15:32.746214 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:32.746176 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:15:33.105728 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:33.105700 2572 scope.go:117] "RemoveContainer" containerID="32cb2f49cf364963c2d79a198a2a122eefc363a22a1e1ac2b381b818ee3a99d3" Jul 14 19:15:33.105918 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:33.105892 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-789b67ddd8-v7lrh_open-cluster-management-agent-addon(e4919425-84e6-4f07-821c-87d4af4e71af)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" podUID="e4919425-84e6-4f07-821c-87d4af4e71af" Jul 14 19:15:34.607349 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:34.607308 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:34.790386 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:34.790302 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:34.970859 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:34.970821 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jhvtj"] Jul 14 19:15:34.997251 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:34.997226 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:34.999523 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:34.999498 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 19:15:34.999638 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:34.999562 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vnvln\"" Jul 14 19:15:34.999638 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:34.999508 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 19:15:34.999829 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:34.999816 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 19:15:35.000590 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.000573 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 19:15:35.000590 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.000583 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 19:15:35.000716 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.000574 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 19:15:35.038095 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.038072 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.038193 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.038116 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-root\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.038193 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.038144 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz6mm\" (UniqueName: \"kubernetes.io/projected/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-kube-api-access-lz6mm\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.038193 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.038171 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-metrics-client-ca\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.038193 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.038188 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-sys\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.038333 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.038205 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.038333 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.038301 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-wtmp\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.038394 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.038337 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-textfile\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.038394 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.038356 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-tls\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.114333 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.114303 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4kwvb" event={"ID":"005d066e-4e27-42af-8e47-e6691b93f148","Type":"ContainerStarted","Data":"effc49c42ee31dbd3a2705434143d609158204895e8b8e2ecb6dfdd2187955e3"} Jul 14 19:15:35.114458 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.114351 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4kwvb" event={"ID":"005d066e-4e27-42af-8e47-e6691b93f148","Type":"ContainerStarted","Data":"38d3da79bffba8aed6f4efe5110a31ef97668d3f39aec6831aa63f8472f44271"} Jul 14 19:15:35.114458 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.114441 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4kwvb" Jul 14 19:15:35.133173 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.133129 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4kwvb" podStartSLOduration=251.288789239 podStartE2EDuration="4m14.133111545s" totalImagesPullingTime="2.844322306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:35.131637771 +0000 UTC m=+284.496708153" watchObservedRunningTime="2026-07-14 19:15:35.133111545 +0000 UTC m=+284.498181906" Jul 14 19:15:35.139561 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.139536 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.139661 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.139590 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-wtmp\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.139661 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.139610 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-textfile\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.139755 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.139736 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-tls\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.139755 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.139746 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-wtmp\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.139854 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.139773 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.139854 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:35.139835 2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 14 19:15:35.139954 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.139869 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-root\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.139954 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:35.139887 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-tls podName:de665ef3-1bee-4c65-b71c-7a35a9afe9e3 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:35.639869402 +0000 UTC m=+285.004939755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-tls") pod "node-exporter-jhvtj" (UID: "de665ef3-1bee-4c65-b71c-7a35a9afe9e3") : secret "node-exporter-tls" not found Jul 14 19:15:35.139954 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.139838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-root\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.139954 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.139907 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-textfile\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.139954 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.139952 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz6mm\" (UniqueName: \"kubernetes.io/projected/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-kube-api-access-lz6mm\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.140217 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.139989 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-metrics-client-ca\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.140217 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.140015 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-sys\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.140217 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.140102 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-sys\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.140389 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.140368 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.140556 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.140537 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-metrics-client-ca\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.142237 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.142217 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.148225 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.148203 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz6mm\" (UniqueName: \"kubernetes.io/projected/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-kube-api-access-lz6mm\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.644130 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:35.644093 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-tls\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:35.644507 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:35.644243 2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 14 19:15:35.644507 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:35.644332 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-tls podName:de665ef3-1bee-4c65-b71c-7a35a9afe9e3 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:36.644316948 +0000 UTC m=+286.009387291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-tls") pod "node-exporter-jhvtj" (UID: "de665ef3-1bee-4c65-b71c-7a35a9afe9e3") : secret "node-exporter-tls" not found Jul 14 19:15:36.650565 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:36.650534 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-tls\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:36.652800 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:36.652770 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de665ef3-1bee-4c65-b71c-7a35a9afe9e3-node-exporter-tls\") pod \"node-exporter-jhvtj\" (UID: \"de665ef3-1bee-4c65-b71c-7a35a9afe9e3\") " pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:36.806745 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:36.806711 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jhvtj" Jul 14 19:15:36.817659 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:15:36.817628 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde665ef3_1bee_4c65_b71c_7a35a9afe9e3.slice/crio-8d243f21a09e07dc283985a6a341a7dbb677639ecfc032d6b0afc4b317c20fda WatchSource:0}: Error finding container 8d243f21a09e07dc283985a6a341a7dbb677639ecfc032d6b0afc4b317c20fda: Status 404 returned error can't find the container with id 8d243f21a09e07dc283985a6a341a7dbb677639ecfc032d6b0afc4b317c20fda Jul 14 19:15:37.125920 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:37.125888 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jhvtj" event={"ID":"de665ef3-1bee-4c65-b71c-7a35a9afe9e3","Type":"ContainerStarted","Data":"8d243f21a09e07dc283985a6a341a7dbb677639ecfc032d6b0afc4b317c20fda"} Jul 14 19:15:38.130508 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:38.130418 2572 generic.go:356] "Generic (PLEG): container finished" podID="de665ef3-1bee-4c65-b71c-7a35a9afe9e3" containerID="0694585b969b24664f8e5b350aef8e3810eb1beb9edb62e177633cf9b884fb61" exitCode=0 Jul 14 19:15:38.130508 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:38.130478 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jhvtj" event={"ID":"de665ef3-1bee-4c65-b71c-7a35a9afe9e3","Type":"ContainerDied","Data":"0694585b969b24664f8e5b350aef8e3810eb1beb9edb62e177633cf9b884fb61"} Jul 14 19:15:38.354901 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:38.354867 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:38.437829 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:38.437742 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:38.556513 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:38.556474 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:39.120215 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:39.120179 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:39.135347 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:39.135316 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jhvtj" event={"ID":"de665ef3-1bee-4c65-b71c-7a35a9afe9e3","Type":"ContainerStarted","Data":"87259142dbf4dc0eecc637886e452bcfd8ae48c57b7cac7c95fb0ce29cb9b109"} Jul 14 19:15:39.135703 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:39.135365 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jhvtj" event={"ID":"de665ef3-1bee-4c65-b71c-7a35a9afe9e3","Type":"ContainerStarted","Data":"5e6b0db9dc30067f90029697bf154d67893584f2203b71dd09dc3a5c71fae66a"} Jul 14 19:15:39.183773 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:39.183737 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-h7v99" podUID="6e91d78e-4060-4063-86ac-5d7be516f269" Jul 14 19:15:39.220995 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:39.220959 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:39.601657 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:39.601617 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:40.115388 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:40.115362 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:41.122649 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:41.122614 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:41.381422 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:41.381325 2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" Jul 14 19:15:41.381662 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:41.381649 2572 scope.go:117] "RemoveContainer" containerID="ef322abd1128b799484dd5f7c4eb939cae34f7bceb39132fe850db1fba99ef95" Jul 14 19:15:41.381807 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:41.381791 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j_open-cluster-management-agent-addon(3542f768-c658-451b-b911-b8956f9e8e39)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" podUID="3542f768-c658-451b-b911-b8956f9e8e39" Jul 14 19:15:41.388653 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:41.388623 2572 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" podUID="fd0d0af9-9b9d-48db-98b3-b7a0bd65987d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 14 19:15:41.388769 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:41.388672 2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" Jul 14 19:15:41.389075 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:41.389029 2572 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"a53b92cba8ca269e6cd9cbac7c90d5ac13c2e9c7e40bd8b32931b8506a933777"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 14 19:15:41.389120 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:41.389092 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" podUID="fd0d0af9-9b9d-48db-98b3-b7a0bd65987d" containerName="service-proxy" containerID="cri-o://a53b92cba8ca269e6cd9cbac7c90d5ac13c2e9c7e40bd8b32931b8506a933777" gracePeriod=30 Jul 14 19:15:41.396789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:41.396751 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jhvtj" podStartSLOduration=6.277216898 podStartE2EDuration="7.396738519s" totalImagesPullingTime="1.053402486s" totalInitContainerRuntime="66.119135ms" isStatefulPod=true podCreationTimestamp="2026-07-14 19:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:39.157270457 +0000 UTC m=+288.522340841" watchObservedRunningTime="2026-07-14 19:15:41.396738519 +0000 UTC m=+290.761808880" Jul 14 19:15:41.431959 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:41.431933 2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:15:41.432319 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:41.432306 2572 scope.go:117] "RemoveContainer" containerID="32cb2f49cf364963c2d79a198a2a122eefc363a22a1e1ac2b381b818ee3a99d3" Jul 14 19:15:41.432478 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:15:41.432459 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-789b67ddd8-v7lrh_open-cluster-management-agent-addon(e4919425-84e6-4f07-821c-87d4af4e71af)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" podUID="e4919425-84e6-4f07-821c-87d4af4e71af" Jul 14 19:15:42.148113 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:42.148078 2572 generic.go:356] "Generic (PLEG): container finished" podID="fd0d0af9-9b9d-48db-98b3-b7a0bd65987d" containerID="a53b92cba8ca269e6cd9cbac7c90d5ac13c2e9c7e40bd8b32931b8506a933777" exitCode=2 Jul 14 19:15:42.148522 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:42.148128 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" event={"ID":"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d","Type":"ContainerDied","Data":"a53b92cba8ca269e6cd9cbac7c90d5ac13c2e9c7e40bd8b32931b8506a933777"} Jul 14 19:15:42.148522 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:42.148162 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dff65c897-qrf6g" event={"ID":"fd0d0af9-9b9d-48db-98b3-b7a0bd65987d","Type":"ContainerStarted","Data":"94b87fd6fa51c93275d498f6e8aed3335392434fdb933cdc1ee9f6d5e8f6a1d1"} Jul 14 19:15:45.120264 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:45.120236 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4kwvb" Jul 14 19:15:45.321884 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:45.321853 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:45.390471 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:45.390380 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:51.013440 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:51.013409 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:15:51.014831 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:51.014800 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:15:51.017156 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:51.017136 2572 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 19:15:51.106830 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:51.106788 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:51.395892 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:51.395859 2572 patch_prober.go:28] interesting pod/image-registry-7f44db5d46-9wzvr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 14 19:15:51.397134 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:51.395908 2572 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" podUID="b2c03a20-3487-4987-b514-d80fbb77da40" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:15:52.307285 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:52.307244 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:52.623939 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:52.623906 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:52.724213 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:52.724172 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:53.109613 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:53.109588 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:15:53.182690 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:53.182664 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:15:53.366975 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:53.366885 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f44db5d46-9wzvr"] Jul 14 19:15:53.381085 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:53.380994 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:53.482669 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:53.482635 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:53.547112 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:53.547072 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:53.771830 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:53.771741 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:53.829612 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:53.829583 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:54.247017 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:54.246985 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:54.311247 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:54.311217 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:55.187415 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:55.187375 2572 scope.go:117] "RemoveContainer" containerID="32cb2f49cf364963c2d79a198a2a122eefc363a22a1e1ac2b381b818ee3a99d3" Jul 14 19:15:55.187857 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:55.187517 2572 scope.go:117] "RemoveContainer" containerID="ef322abd1128b799484dd5f7c4eb939cae34f7bceb39132fe850db1fba99ef95" Jul 14 19:15:56.193535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:56.193491 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d59cd6fc4-bc69j" event={"ID":"3542f768-c658-451b-b911-b8956f9e8e39","Type":"ContainerStarted","Data":"56fd647c4fe53b66b19a682a7e8a1ebaa8329b4b8c142a30016afc6db58e536c"} Jul 14 19:15:56.195316 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:56.195291 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" event={"ID":"e4919425-84e6-4f07-821c-87d4af4e71af","Type":"ContainerStarted","Data":"6f6bc7df800789a0c364cc1d02b8563138f62ce95c1fd43b2fb66b24781c0123"} Jul 14 19:15:56.195636 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:56.195616 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:15:56.197232 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:56.197211 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-789b67ddd8-v7lrh" Jul 14 19:15:57.060307 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:57.060276 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:15:57.161318 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:15:57.161286 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:01.712314 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:01.712268 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:01.812416 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:01.812383 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:02.196062 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:02.196021 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-95h2r_b4b0f202-374a-4e71-97db-50a6422c7767/serve-healthcheck-canary/0.log" Jul 14 19:16:10.774811 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:10.774771 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:10.875683 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:10.875647 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:12.266270 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:12.266198 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:12.329223 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:12.329174 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:13.080499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:13.080444 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:13.136114 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:13.136079 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:18.387686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.387654 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" podUID="b2c03a20-3487-4987-b514-d80fbb77da40" containerName="registry" containerID="cri-o://f95f1eaaa2b9143b8efe07079976b267c5c88fbbb0a8569d5e33b9d26e7bf842" gracePeriod=30 Jul 14 19:16:18.626367 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.626344 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:16:18.768587 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.768498 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2c03a20-3487-4987-b514-d80fbb77da40-trusted-ca\") pod \"b2c03a20-3487-4987-b514-d80fbb77da40\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " Jul 14 19:16:18.768587 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.768556 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4qqw\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-kube-api-access-r4qqw\") pod \"b2c03a20-3487-4987-b514-d80fbb77da40\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " Jul 14 19:16:18.768587 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.768588 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2c03a20-3487-4987-b514-d80fbb77da40-image-registry-private-configuration\") pod \"b2c03a20-3487-4987-b514-d80fbb77da40\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " Jul 14 19:16:18.768854 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.768617 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2c03a20-3487-4987-b514-d80fbb77da40-registry-certificates\") pod \"b2c03a20-3487-4987-b514-d80fbb77da40\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " Jul 14 19:16:18.768854 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.768636 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2c03a20-3487-4987-b514-d80fbb77da40-ca-trust-extracted\") pod \"b2c03a20-3487-4987-b514-d80fbb77da40\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " Jul 14 19:16:18.768854 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.768657 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls\") pod \"b2c03a20-3487-4987-b514-d80fbb77da40\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " Jul 14 19:16:18.768854 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.768681 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-bound-sa-token\") pod \"b2c03a20-3487-4987-b514-d80fbb77da40\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " Jul 14 19:16:18.768854 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.768707 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2c03a20-3487-4987-b514-d80fbb77da40-installation-pull-secrets\") pod \"b2c03a20-3487-4987-b514-d80fbb77da40\" (UID: \"b2c03a20-3487-4987-b514-d80fbb77da40\") " Jul 14 19:16:18.769188 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.769124 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c03a20-3487-4987-b514-d80fbb77da40-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b2c03a20-3487-4987-b514-d80fbb77da40" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:16:18.769320 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.769276 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c03a20-3487-4987-b514-d80fbb77da40-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b2c03a20-3487-4987-b514-d80fbb77da40" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:16:18.771260 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.771206 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c03a20-3487-4987-b514-d80fbb77da40-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b2c03a20-3487-4987-b514-d80fbb77da40" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:18.771420 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.771390 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b2c03a20-3487-4987-b514-d80fbb77da40" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:16:18.771499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.771429 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c03a20-3487-4987-b514-d80fbb77da40-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b2c03a20-3487-4987-b514-d80fbb77da40" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:18.771499 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.771488 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-kube-api-access-r4qqw" (OuterVolumeSpecName: "kube-api-access-r4qqw") pod "b2c03a20-3487-4987-b514-d80fbb77da40" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40"). InnerVolumeSpecName "kube-api-access-r4qqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:16:18.771592 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.771514 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b2c03a20-3487-4987-b514-d80fbb77da40" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:16:18.781215 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.781172 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c03a20-3487-4987-b514-d80fbb77da40-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b2c03a20-3487-4987-b514-d80fbb77da40" (UID: "b2c03a20-3487-4987-b514-d80fbb77da40"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:16:18.870055 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.870018 2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2c03a20-3487-4987-b514-d80fbb77da40-registry-certificates\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:16:18.870207 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.870068 2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2c03a20-3487-4987-b514-d80fbb77da40-ca-trust-extracted\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:16:18.870207 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.870079 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-registry-tls\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:16:18.870207 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.870089 2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-bound-sa-token\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:16:18.870207 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.870097 2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2c03a20-3487-4987-b514-d80fbb77da40-installation-pull-secrets\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:16:18.870207 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.870105 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2c03a20-3487-4987-b514-d80fbb77da40-trusted-ca\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:16:18.870207 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.870113 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4qqw\" (UniqueName: \"kubernetes.io/projected/b2c03a20-3487-4987-b514-d80fbb77da40-kube-api-access-r4qqw\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:16:18.870207 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:18.870123 2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2c03a20-3487-4987-b514-d80fbb77da40-image-registry-private-configuration\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:16:19.209577 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:19.209541 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:19.271885 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:19.271853 2572 generic.go:356] "Generic (PLEG): container finished" podID="b2c03a20-3487-4987-b514-d80fbb77da40" containerID="f95f1eaaa2b9143b8efe07079976b267c5c88fbbb0a8569d5e33b9d26e7bf842" exitCode=0 Jul 14 19:16:19.272023 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:19.271903 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" Jul 14 19:16:19.272023 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:19.271940 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" event={"ID":"b2c03a20-3487-4987-b514-d80fbb77da40","Type":"ContainerDied","Data":"f95f1eaaa2b9143b8efe07079976b267c5c88fbbb0a8569d5e33b9d26e7bf842"} Jul 14 19:16:19.272023 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:19.271993 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f44db5d46-9wzvr" event={"ID":"b2c03a20-3487-4987-b514-d80fbb77da40","Type":"ContainerDied","Data":"1c2a9514ac60e222122593b50e3d1a656c0f28e6beb60d44744a9d7d980b53ce"} Jul 14 19:16:19.272023 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:19.272013 2572 scope.go:117] "RemoveContainer" containerID="f95f1eaaa2b9143b8efe07079976b267c5c88fbbb0a8569d5e33b9d26e7bf842" Jul 14 19:16:19.281614 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:19.281581 2572 scope.go:117] "RemoveContainer" containerID="f95f1eaaa2b9143b8efe07079976b267c5c88fbbb0a8569d5e33b9d26e7bf842" Jul 14 19:16:19.281923 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:16:19.281902 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f95f1eaaa2b9143b8efe07079976b267c5c88fbbb0a8569d5e33b9d26e7bf842\": container with ID starting with f95f1eaaa2b9143b8efe07079976b267c5c88fbbb0a8569d5e33b9d26e7bf842 not found: ID does not exist" containerID="f95f1eaaa2b9143b8efe07079976b267c5c88fbbb0a8569d5e33b9d26e7bf842" Jul 14 19:16:19.281989 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:19.281932 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f95f1eaaa2b9143b8efe07079976b267c5c88fbbb0a8569d5e33b9d26e7bf842"} err="failed to get container status \"f95f1eaaa2b9143b8efe07079976b267c5c88fbbb0a8569d5e33b9d26e7bf842\": rpc error: code = NotFound desc = could not find container \"f95f1eaaa2b9143b8efe07079976b267c5c88fbbb0a8569d5e33b9d26e7bf842\": container with ID starting with f95f1eaaa2b9143b8efe07079976b267c5c88fbbb0a8569d5e33b9d26e7bf842 not found: ID does not exist" Jul 14 19:16:19.286401 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:19.286381 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:19.293322 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:19.293305 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f44db5d46-9wzvr"] Jul 14 19:16:19.297072 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:19.297034 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f44db5d46-9wzvr"] Jul 14 19:16:21.187392 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:21.187362 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2c03a20-3487-4987-b514-d80fbb77da40" path="/var/lib/kubelet/pods/b2c03a20-3487-4987-b514-d80fbb77da40/volumes" Jul 14 19:16:25.781671 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:25.781636 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:26.792329 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:26.792280 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:33.598917 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:33.598886 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:33.699455 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:33.699396 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:33.760900 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:33.760854 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:34.452914 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:34.452877 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:34.515751 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:34.515717 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:54.140378 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:54.140319 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:54.376639 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:54.376593 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:16:59.241070 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.241030 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8576574855-l8cgz"] Jul 14 19:16:59.241459 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.241275 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c03a20-3487-4987-b514-d80fbb77da40" containerName="registry" Jul 14 19:16:59.241459 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.241286 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c03a20-3487-4987-b514-d80fbb77da40" containerName="registry" Jul 14 19:16:59.241459 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.241333 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2c03a20-3487-4987-b514-d80fbb77da40" containerName="registry" Jul 14 19:16:59.245424 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.245408 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.248410 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.248384 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 14 19:16:59.248548 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.248477 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rghpn\"" Jul 14 19:16:59.248548 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.248491 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 14 19:16:59.248548 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.248497 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 14 19:16:59.248685 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.248607 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 14 19:16:59.249073 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.249059 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 14 19:16:59.254554 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.254533 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 14 19:16:59.261465 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.261445 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8576574855-l8cgz"] Jul 14 19:16:59.352681 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.352642 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e979de95-0f60-4ef1-868f-6ec4a12444f3-secret-telemeter-client\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.352681 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.352681 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e979de95-0f60-4ef1-868f-6ec4a12444f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.352943 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.352704 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e979de95-0f60-4ef1-868f-6ec4a12444f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.352943 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.352806 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e979de95-0f60-4ef1-868f-6ec4a12444f3-metrics-client-ca\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.352943 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.352854 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6wdk\" (UniqueName: \"kubernetes.io/projected/e979de95-0f60-4ef1-868f-6ec4a12444f3-kube-api-access-s6wdk\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.352943 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.352905 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e979de95-0f60-4ef1-868f-6ec4a12444f3-serving-certs-ca-bundle\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.352943 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.352930 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e979de95-0f60-4ef1-868f-6ec4a12444f3-telemeter-client-tls\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.353153 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.352982 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e979de95-0f60-4ef1-868f-6ec4a12444f3-federate-client-tls\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.454061 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.454008 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e979de95-0f60-4ef1-868f-6ec4a12444f3-secret-telemeter-client\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.454061 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.454063 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e979de95-0f60-4ef1-868f-6ec4a12444f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.454284 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.454098 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e979de95-0f60-4ef1-868f-6ec4a12444f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.454284 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.454124 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e979de95-0f60-4ef1-868f-6ec4a12444f3-metrics-client-ca\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.454284 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.454147 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6wdk\" (UniqueName: \"kubernetes.io/projected/e979de95-0f60-4ef1-868f-6ec4a12444f3-kube-api-access-s6wdk\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.454284 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.454166 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e979de95-0f60-4ef1-868f-6ec4a12444f3-serving-certs-ca-bundle\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.454284 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.454184 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e979de95-0f60-4ef1-868f-6ec4a12444f3-telemeter-client-tls\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.454284 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.454207 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e979de95-0f60-4ef1-868f-6ec4a12444f3-federate-client-tls\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.454988 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.454957 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e979de95-0f60-4ef1-868f-6ec4a12444f3-metrics-client-ca\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.455160 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.454965 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e979de95-0f60-4ef1-868f-6ec4a12444f3-serving-certs-ca-bundle\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.455233 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.455168 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e979de95-0f60-4ef1-868f-6ec4a12444f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.456706 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.456679 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e979de95-0f60-4ef1-868f-6ec4a12444f3-federate-client-tls\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.456823 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.456730 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e979de95-0f60-4ef1-868f-6ec4a12444f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.456986 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.456969 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e979de95-0f60-4ef1-868f-6ec4a12444f3-telemeter-client-tls\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.457200 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.457184 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e979de95-0f60-4ef1-868f-6ec4a12444f3-secret-telemeter-client\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.463209 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.463182 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6wdk\" (UniqueName: \"kubernetes.io/projected/e979de95-0f60-4ef1-868f-6ec4a12444f3-kube-api-access-s6wdk\") pod \"telemeter-client-8576574855-l8cgz\" (UID: \"e979de95-0f60-4ef1-868f-6ec4a12444f3\") " pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.554967 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.554888 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" Jul 14 19:16:59.690706 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.688956 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8576574855-l8cgz"] Jul 14 19:16:59.693634 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:16:59.693595 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode979de95_0f60_4ef1_868f_6ec4a12444f3.slice/crio-80e8f101326eef313f22109634de4d5e161bf564f8367efe7505e70256b66af8 WatchSource:0}: Error finding container 80e8f101326eef313f22109634de4d5e161bf564f8367efe7505e70256b66af8: Status 404 returned error can't find the container with id 80e8f101326eef313f22109634de4d5e161bf564f8367efe7505e70256b66af8 Jul 14 19:16:59.695548 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:16:59.695532 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:17:00.403334 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:00.403298 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" event={"ID":"e979de95-0f60-4ef1-868f-6ec4a12444f3","Type":"ContainerStarted","Data":"80e8f101326eef313f22109634de4d5e161bf564f8367efe7505e70256b66af8"} Jul 14 19:17:02.411348 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:02.411318 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" event={"ID":"e979de95-0f60-4ef1-868f-6ec4a12444f3","Type":"ContainerStarted","Data":"54d8b573da912c229c337c5405e2829ab1180f7a64277f412ac8bc096ce6c15d"} Jul 14 19:17:02.881190 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:02.881150 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:17:03.416749 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:03.416710 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" event={"ID":"e979de95-0f60-4ef1-868f-6ec4a12444f3","Type":"ContainerStarted","Data":"0b39686f4bbe755d3efeddee4cf7e9d161e32fa6800e969aceb7f0918368a059"} Jul 14 19:17:03.417148 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:03.416764 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" event={"ID":"e979de95-0f60-4ef1-868f-6ec4a12444f3","Type":"ContainerStarted","Data":"fa451442fd299ec9713dca63406b7ac15fe7b41c455b3b4e4a8f95bed7f42961"} Jul 14 19:17:03.453656 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:03.453601 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8576574855-l8cgz" podStartSLOduration=1.6702611840000001 podStartE2EDuration="4.453588638s" totalImagesPullingTime="2.783327454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:16:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:17:03.4523785 +0000 UTC m=+372.817448862" watchObservedRunningTime="2026-07-14 19:17:03.453588638 +0000 UTC m=+372.818658999" Jul 14 19:17:04.999852 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:04.999797 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:17:05.002156 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:05.002131 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e91d78e-4060-4063-86ac-5d7be516f269-metrics-certs\") pod \"network-metrics-daemon-h7v99\" (UID: \"6e91d78e-4060-4063-86ac-5d7be516f269\") " pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:17:05.185628 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:05.185596 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xxpfz\"" Jul 14 19:17:05.193713 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:05.193693 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7v99" Jul 14 19:17:05.335223 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:05.335200 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h7v99"] Jul 14 19:17:05.336964 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:17:05.336943 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e91d78e_4060_4063_86ac_5d7be516f269.slice/crio-375b218057c808791bf28622fc73ae1fa62d0901616b3ac0e6d608d5baf1c761 WatchSource:0}: Error finding container 375b218057c808791bf28622fc73ae1fa62d0901616b3ac0e6d608d5baf1c761: Status 404 returned error can't find the container with id 375b218057c808791bf28622fc73ae1fa62d0901616b3ac0e6d608d5baf1c761 Jul 14 19:17:05.424364 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:05.424326 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7v99" event={"ID":"6e91d78e-4060-4063-86ac-5d7be516f269","Type":"ContainerStarted","Data":"375b218057c808791bf28622fc73ae1fa62d0901616b3ac0e6d608d5baf1c761"} Jul 14 19:17:06.199922 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:06.199892 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:17:06.301612 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:06.301581 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:17:06.429812 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:06.429722 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7v99" event={"ID":"6e91d78e-4060-4063-86ac-5d7be516f269","Type":"ContainerStarted","Data":"833e9cab1ca7f78f52f7c2c2d2f2c69f2b10b2feb11ed9a1833ce6649bc71b19"} Jul 14 19:17:06.429812 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:06.429778 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7v99" event={"ID":"6e91d78e-4060-4063-86ac-5d7be516f269","Type":"ContainerStarted","Data":"959d2d594ed6913eee11d715a3ea0acce9f144dfcf5c76138dccbab9f91f3218"} Jul 14 19:17:06.446948 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:06.446899 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h7v99" podStartSLOduration=374.63682162 podStartE2EDuration="6m15.44688439s" totalImagesPullingTime="810.06277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:17:06.445573999 +0000 UTC m=+375.810644363" watchObservedRunningTime="2026-07-14 19:17:06.44688439 +0000 UTC m=+375.811954751" Jul 14 19:17:09.006288 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:09.006249 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:17:09.115363 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:09.115335 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:17:13.407377 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:13.407346 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:17:13.469223 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:13.469180 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:17:15.446658 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:15.446625 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:17:15.552117 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:15.552084 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:17:15.654740 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:15.654707 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:17:32.506535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:32.506487 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:17:32.766613 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:17:32.766524 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:01.142813 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:01.142777 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:01.227191 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:01.227153 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:09.599883 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:09.599839 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:09.720903 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:09.720862 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:11.177791 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:11.177756 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:11.302552 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:11.302520 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:11.405307 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:11.405270 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:11.922392 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:11.922346 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:12.024412 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:12.024382 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:21.370135 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:21.370103 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:21.433981 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:21.433947 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:21.732616 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:21.732533 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:21.832848 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:21.832819 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:21.896691 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:21.896660 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:25.789023 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:25.788986 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:26.502664 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:26.502619 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:18:26.604354 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:18:26.604304 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:19:09.638025 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:19:09.637986 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:19:16.002680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:19:16.002643 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:19:31.319244 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:19:31.319206 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:19:31.418919 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:19:31.418884 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:19:58.075909 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:19:58.075819 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:19:58.907614 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:19:58.907578 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:01.669598 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:01.669558 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:01.771310 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:01.771272 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:03.484396 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:03.484360 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:03.636282 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:03.636239 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:03.869705 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:03.869670 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:06.057787 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:06.057743 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:06.260447 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:06.260411 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:07.780643 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:07.780596 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:08.477542 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:08.477506 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:08.578027 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:08.577989 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:08.678582 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:08.678536 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:12.854031 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:12.853994 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:12.936010 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:12.935972 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:13.652294 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:13.652252 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:14.089348 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:14.089315 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ddp9v"] Jul 14 19:20:14.092168 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:14.092149 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ddp9v" Jul 14 19:20:14.094727 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:14.094703 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 14 19:20:14.094820 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:14.094740 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-4c99v\"" Jul 14 19:20:14.095598 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:14.095583 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 14 19:20:14.101830 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:14.101812 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ddp9v"] Jul 14 19:20:14.139865 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:14.139841 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1356680a-47fd-4836-9fe1-fd9c2d24cb93-bound-sa-token\") pod \"cert-manager-55b4d954d5-ddp9v\" (UID: \"1356680a-47fd-4836-9fe1-fd9c2d24cb93\") " pod="cert-manager/cert-manager-55b4d954d5-ddp9v" Jul 14 19:20:14.139956 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:14.139896 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkj8j\" (UniqueName: \"kubernetes.io/projected/1356680a-47fd-4836-9fe1-fd9c2d24cb93-kube-api-access-mkj8j\") pod \"cert-manager-55b4d954d5-ddp9v\" (UID: \"1356680a-47fd-4836-9fe1-fd9c2d24cb93\") " pod="cert-manager/cert-manager-55b4d954d5-ddp9v" Jul 14 19:20:14.240818 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:14.240788 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkj8j\" (UniqueName: \"kubernetes.io/projected/1356680a-47fd-4836-9fe1-fd9c2d24cb93-kube-api-access-mkj8j\") pod \"cert-manager-55b4d954d5-ddp9v\" (UID: \"1356680a-47fd-4836-9fe1-fd9c2d24cb93\") " pod="cert-manager/cert-manager-55b4d954d5-ddp9v" Jul 14 19:20:14.240974 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:14.240829 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1356680a-47fd-4836-9fe1-fd9c2d24cb93-bound-sa-token\") pod \"cert-manager-55b4d954d5-ddp9v\" (UID: \"1356680a-47fd-4836-9fe1-fd9c2d24cb93\") " pod="cert-manager/cert-manager-55b4d954d5-ddp9v" Jul 14 19:20:14.249682 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:14.249656 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1356680a-47fd-4836-9fe1-fd9c2d24cb93-bound-sa-token\") pod \"cert-manager-55b4d954d5-ddp9v\" (UID: \"1356680a-47fd-4836-9fe1-fd9c2d24cb93\") " pod="cert-manager/cert-manager-55b4d954d5-ddp9v" Jul 14 19:20:14.249781 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:14.249709 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkj8j\" (UniqueName: \"kubernetes.io/projected/1356680a-47fd-4836-9fe1-fd9c2d24cb93-kube-api-access-mkj8j\") pod \"cert-manager-55b4d954d5-ddp9v\" (UID: \"1356680a-47fd-4836-9fe1-fd9c2d24cb93\") " pod="cert-manager/cert-manager-55b4d954d5-ddp9v" Jul 14 19:20:14.400848 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:14.400756 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ddp9v" Jul 14 19:20:14.521374 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:14.521209 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ddp9v"] Jul 14 19:20:14.523301 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:20:14.523267 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1356680a_47fd_4836_9fe1_fd9c2d24cb93.slice/crio-b3cc13cc09f30869738db417ba1f09c2d7059c61cab5077ee406a8376e221338 WatchSource:0}: Error finding container b3cc13cc09f30869738db417ba1f09c2d7059c61cab5077ee406a8376e221338: Status 404 returned error can't find the container with id b3cc13cc09f30869738db417ba1f09c2d7059c61cab5077ee406a8376e221338 Jul 14 19:20:15.046083 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:15.046024 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ddp9v" event={"ID":"1356680a-47fd-4836-9fe1-fd9c2d24cb93","Type":"ContainerStarted","Data":"b3cc13cc09f30869738db417ba1f09c2d7059c61cab5077ee406a8376e221338"} Jul 14 19:20:18.059891 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:18.059856 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ddp9v" event={"ID":"1356680a-47fd-4836-9fe1-fd9c2d24cb93","Type":"ContainerStarted","Data":"fb64cf596925c2a728dde3639f6b82e12508370465a14211025d506746969bc4"} Jul 14 19:20:18.083128 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:18.083081 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ddp9v" podStartSLOduration=1.323023874 podStartE2EDuration="4.083066942s" totalImagesPullingTime="2.760043068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:20:18.081295128 +0000 UTC m=+567.446365500" watchObservedRunningTime="2026-07-14 19:20:18.083066942 +0000 UTC m=+567.448137300" Jul 14 19:20:21.183193 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:21.183158 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:26.521679 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:26.521636 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:26.644203 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:26.644154 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:31.945638 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:31.945603 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:34.252283 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:34.252244 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:34.416732 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:34.416702 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:40.929527 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:40.929488 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:41.057854 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:41.057821 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:50.126559 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:50.126518 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:50.292814 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:50.292779 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:50.710798 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:50.710767 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:51.009512 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.009418 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:51.035132 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.035103 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:20:51.035519 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.035504 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:20:51.149274 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.149245 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz"] Jul 14 19:20:51.152213 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.152198 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.156681 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.156660 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 19:20:51.156820 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.156702 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 19:20:51.156889 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.156844 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2r59f\"" Jul 14 19:20:51.156955 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.156913 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 19:20:51.157009 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.156975 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 19:20:51.169110 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.169084 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz"] Jul 14 19:20:51.206451 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.206428 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.206565 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.206471 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.206565 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.206519 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-tmp\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.206565 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.206537 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfc6b\" (UniqueName: \"kubernetes.io/projected/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-kube-api-access-bfc6b\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.206670 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.206574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-manifests\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.307412 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.307330 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.307412 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.307370 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.307412 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.307405 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-tmp\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.307657 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.307425 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfc6b\" (UniqueName: \"kubernetes.io/projected/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-kube-api-access-bfc6b\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.307657 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.307459 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-manifests\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.307815 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.307792 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-manifests\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.309651 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.309625 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-tmp\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.309915 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.309896 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.309986 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.309942 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.319422 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.319398 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfc6b\" (UniqueName: \"kubernetes.io/projected/94ff3e11-5fd0-4d91-8268-ca0353d8bda8-kube-api-access-bfc6b\") pod \"opendatahub-operator-controller-manager-6f9786d97d-c99zz\" (UID: \"94ff3e11-5fd0-4d91-8268-ca0353d8bda8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.331892 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.331857 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:51.462421 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.462303 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:51.591242 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.591216 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz"] Jul 14 19:20:51.593645 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:20:51.593618 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ff3e11_5fd0_4d91_8268_ca0353d8bda8.slice/crio-5581b1b6e234b78f3b78d1fc0cb358666d7ea149fe6af7dc724458c177c173bf WatchSource:0}: Error finding container 5581b1b6e234b78f3b78d1fc0cb358666d7ea149fe6af7dc724458c177c173bf: Status 404 returned error can't find the container with id 5581b1b6e234b78f3b78d1fc0cb358666d7ea149fe6af7dc724458c177c173bf Jul 14 19:20:51.681371 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.681337 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:51.786242 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:51.786211 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:52.174853 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:52.174822 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" event={"ID":"94ff3e11-5fd0-4d91-8268-ca0353d8bda8","Type":"ContainerStarted","Data":"5581b1b6e234b78f3b78d1fc0cb358666d7ea149fe6af7dc724458c177c173bf"} Jul 14 19:20:52.290721 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:52.290682 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:55.890792 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:55.890755 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:20:57.194636 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:57.194597 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" event={"ID":"94ff3e11-5fd0-4d91-8268-ca0353d8bda8","Type":"ContainerStarted","Data":"85584c8dc42b57b99f62c0694eec57c00cf49bc7704b11a39d2ad4fc77466005"} Jul 14 19:20:58.199316 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:58.199278 2572 generic.go:356] "Generic (PLEG): container finished" podID="94ff3e11-5fd0-4d91-8268-ca0353d8bda8" containerID="85584c8dc42b57b99f62c0694eec57c00cf49bc7704b11a39d2ad4fc77466005" exitCode=0 Jul 14 19:20:58.199721 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:58.199372 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" event={"ID":"94ff3e11-5fd0-4d91-8268-ca0353d8bda8","Type":"ContainerDied","Data":"85584c8dc42b57b99f62c0694eec57c00cf49bc7704b11a39d2ad4fc77466005"} Jul 14 19:20:59.205341 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:59.205308 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" event={"ID":"94ff3e11-5fd0-4d91-8268-ca0353d8bda8","Type":"ContainerStarted","Data":"331a360ef0fdf57ed5d69e419f9a5695210cd544d551eb2e76959c7598d4c0c2"} Jul 14 19:20:59.205739 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:59.205463 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:20:59.229299 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:20:59.229245 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" podStartSLOduration=1.799713648 podStartE2EDuration="8.229227474s" totalImagesPullingTime="5.357025632s" totalInitContainerRuntime="1.072488194s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:20:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:20:59.227402874 +0000 UTC m=+608.592473216" watchObservedRunningTime="2026-07-14 19:20:59.229227474 +0000 UTC m=+608.594297834" Jul 14 19:21:10.211506 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:10.211473 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-c99zz" Jul 14 19:21:11.965388 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:11.965351 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:12.210590 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.210514 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:12.463291 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.463252 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:12.648911 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.648878 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:12.682648 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.682617 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d877944c-8glz9"] Jul 14 19:21:12.786529 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.786445 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d877944c-8glz9"] Jul 14 19:21:12.786529 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.786487 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9jbtj"] Jul 14 19:21:12.786702 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.786572 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d877944c-8glz9" Jul 14 19:21:12.789365 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.789323 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 14 19:21:12.789512 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.789420 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 14 19:21:12.789512 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.789325 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9tf92\"" Jul 14 19:21:12.789512 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.789420 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 19:21:12.789708 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.789686 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:12.790267 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.790249 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 19:21:12.800336 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.800314 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9jbtj"] Jul 14 19:21:12.800447 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.800411 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9jbtj" Jul 14 19:21:12.802668 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.802644 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 14 19:21:12.802921 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.802908 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-n4pdc\"" Jul 14 19:21:12.878165 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.878130 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ff72c66-3b21-4ae2-927b-dea2c6531637-cert\") pod \"odh-model-controller-7998c67dc4-9jbtj\" (UID: \"4ff72c66-3b21-4ae2-927b-dea2c6531637\") " pod="opendatahub/odh-model-controller-7998c67dc4-9jbtj" Jul 14 19:21:12.878165 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.878165 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4eb12cf4-fd60-4976-a21b-86e9d1f6628d-tmp\") pod \"kube-auth-proxy-5d877944c-8glz9\" (UID: \"4eb12cf4-fd60-4976-a21b-86e9d1f6628d\") " pod="openshift-ingress/kube-auth-proxy-5d877944c-8glz9" Jul 14 19:21:12.878392 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.878194 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb12cf4-fd60-4976-a21b-86e9d1f6628d-tls-certs\") pod \"kube-auth-proxy-5d877944c-8glz9\" (UID: \"4eb12cf4-fd60-4976-a21b-86e9d1f6628d\") " pod="openshift-ingress/kube-auth-proxy-5d877944c-8glz9" Jul 14 19:21:12.878392 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.878219 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd65w\" (UniqueName: \"kubernetes.io/projected/4eb12cf4-fd60-4976-a21b-86e9d1f6628d-kube-api-access-vd65w\") pod \"kube-auth-proxy-5d877944c-8glz9\" (UID: \"4eb12cf4-fd60-4976-a21b-86e9d1f6628d\") " pod="openshift-ingress/kube-auth-proxy-5d877944c-8glz9" Jul 14 19:21:12.878392 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.878251 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwbzk\" (UniqueName: \"kubernetes.io/projected/4ff72c66-3b21-4ae2-927b-dea2c6531637-kube-api-access-gwbzk\") pod \"odh-model-controller-7998c67dc4-9jbtj\" (UID: \"4ff72c66-3b21-4ae2-927b-dea2c6531637\") " pod="opendatahub/odh-model-controller-7998c67dc4-9jbtj" Jul 14 19:21:12.892650 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.892617 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:12.978935 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.978905 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ff72c66-3b21-4ae2-927b-dea2c6531637-cert\") pod \"odh-model-controller-7998c67dc4-9jbtj\" (UID: \"4ff72c66-3b21-4ae2-927b-dea2c6531637\") " pod="opendatahub/odh-model-controller-7998c67dc4-9jbtj" Jul 14 19:21:12.978935 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.978939 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4eb12cf4-fd60-4976-a21b-86e9d1f6628d-tmp\") pod \"kube-auth-proxy-5d877944c-8glz9\" (UID: \"4eb12cf4-fd60-4976-a21b-86e9d1f6628d\") " pod="openshift-ingress/kube-auth-proxy-5d877944c-8glz9" Jul 14 19:21:12.979410 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.978976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb12cf4-fd60-4976-a21b-86e9d1f6628d-tls-certs\") pod \"kube-auth-proxy-5d877944c-8glz9\" (UID: \"4eb12cf4-fd60-4976-a21b-86e9d1f6628d\") " pod="openshift-ingress/kube-auth-proxy-5d877944c-8glz9" Jul 14 19:21:12.979410 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.979006 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd65w\" (UniqueName: \"kubernetes.io/projected/4eb12cf4-fd60-4976-a21b-86e9d1f6628d-kube-api-access-vd65w\") pod \"kube-auth-proxy-5d877944c-8glz9\" (UID: \"4eb12cf4-fd60-4976-a21b-86e9d1f6628d\") " pod="openshift-ingress/kube-auth-proxy-5d877944c-8glz9" Jul 14 19:21:12.979410 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.979035 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwbzk\" (UniqueName: \"kubernetes.io/projected/4ff72c66-3b21-4ae2-927b-dea2c6531637-kube-api-access-gwbzk\") pod \"odh-model-controller-7998c67dc4-9jbtj\" (UID: \"4ff72c66-3b21-4ae2-927b-dea2c6531637\") " pod="opendatahub/odh-model-controller-7998c67dc4-9jbtj" Jul 14 19:21:12.981340 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.981315 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4eb12cf4-fd60-4976-a21b-86e9d1f6628d-tmp\") pod \"kube-auth-proxy-5d877944c-8glz9\" (UID: \"4eb12cf4-fd60-4976-a21b-86e9d1f6628d\") " pod="openshift-ingress/kube-auth-proxy-5d877944c-8glz9" Jul 14 19:21:12.981452 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.981371 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ff72c66-3b21-4ae2-927b-dea2c6531637-cert\") pod \"odh-model-controller-7998c67dc4-9jbtj\" (UID: \"4ff72c66-3b21-4ae2-927b-dea2c6531637\") " pod="opendatahub/odh-model-controller-7998c67dc4-9jbtj" Jul 14 19:21:12.981452 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.981415 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb12cf4-fd60-4976-a21b-86e9d1f6628d-tls-certs\") pod \"kube-auth-proxy-5d877944c-8glz9\" (UID: \"4eb12cf4-fd60-4976-a21b-86e9d1f6628d\") " pod="openshift-ingress/kube-auth-proxy-5d877944c-8glz9" Jul 14 19:21:12.990507 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.990487 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwbzk\" (UniqueName: \"kubernetes.io/projected/4ff72c66-3b21-4ae2-927b-dea2c6531637-kube-api-access-gwbzk\") pod \"odh-model-controller-7998c67dc4-9jbtj\" (UID: \"4ff72c66-3b21-4ae2-927b-dea2c6531637\") " pod="opendatahub/odh-model-controller-7998c67dc4-9jbtj" Jul 14 19:21:12.990606 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:12.990519 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd65w\" (UniqueName: \"kubernetes.io/projected/4eb12cf4-fd60-4976-a21b-86e9d1f6628d-kube-api-access-vd65w\") pod \"kube-auth-proxy-5d877944c-8glz9\" (UID: \"4eb12cf4-fd60-4976-a21b-86e9d1f6628d\") " pod="openshift-ingress/kube-auth-proxy-5d877944c-8glz9" Jul 14 19:21:13.002475 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:13.002448 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:13.096442 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:13.096412 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d877944c-8glz9" Jul 14 19:21:13.111189 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:13.111161 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9jbtj" Jul 14 19:21:13.239565 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:13.239540 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d877944c-8glz9"] Jul 14 19:21:13.241752 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:21:13.241728 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb12cf4_fd60_4976_a21b_86e9d1f6628d.slice/crio-787ff51bf51f3cdfe9001219b9efbd45efd23a28914c0696679ea671184d005d WatchSource:0}: Error finding container 787ff51bf51f3cdfe9001219b9efbd45efd23a28914c0696679ea671184d005d: Status 404 returned error can't find the container with id 787ff51bf51f3cdfe9001219b9efbd45efd23a28914c0696679ea671184d005d Jul 14 19:21:13.253854 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:13.253783 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9jbtj"] Jul 14 19:21:13.255700 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:13.255679 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d877944c-8glz9" event={"ID":"4eb12cf4-fd60-4976-a21b-86e9d1f6628d","Type":"ContainerStarted","Data":"787ff51bf51f3cdfe9001219b9efbd45efd23a28914c0696679ea671184d005d"} Jul 14 19:21:13.256274 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:21:13.256253 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff72c66_3b21_4ae2_927b_dea2c6531637.slice/crio-a3160038c392d1c5d551f5bcd6c45b9b48e96e0b35c734b407f71a87c4d1c594 WatchSource:0}: Error finding container a3160038c392d1c5d551f5bcd6c45b9b48e96e0b35c734b407f71a87c4d1c594: Status 404 returned error can't find the container with id a3160038c392d1c5d551f5bcd6c45b9b48e96e0b35c734b407f71a87c4d1c594 Jul 14 19:21:13.626237 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:13.626192 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:13.946437 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:13.946349 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:14.080880 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:14.080829 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:14.262376 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:14.262283 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9jbtj" event={"ID":"4ff72c66-3b21-4ae2-927b-dea2c6531637","Type":"ContainerStarted","Data":"a3160038c392d1c5d551f5bcd6c45b9b48e96e0b35c734b407f71a87c4d1c594"} Jul 14 19:21:14.545633 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:14.545508 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:14.910909 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:14.910856 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:17.276169 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:17.276075 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d877944c-8glz9" event={"ID":"4eb12cf4-fd60-4976-a21b-86e9d1f6628d","Type":"ContainerStarted","Data":"d4c79cac36236114bbd78ab591aac99eaf4f6692aa7ae9d76677cb23d5f2acc2"} Jul 14 19:21:17.277432 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:17.277406 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9jbtj" event={"ID":"4ff72c66-3b21-4ae2-927b-dea2c6531637","Type":"ContainerStarted","Data":"83bfa2da5275e45842643af4abd5a1117f19690021ccfae4e55b8e3a641bb85e"} Jul 14 19:21:17.277560 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:17.277536 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9jbtj" Jul 14 19:21:17.292907 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:17.292865 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d877944c-8glz9" podStartSLOduration=1.6325074910000001 podStartE2EDuration="5.292851473s" totalImagesPullingTime="3.660343982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:21:17.291846506 +0000 UTC m=+626.656916866" watchObservedRunningTime="2026-07-14 19:21:17.292851473 +0000 UTC m=+626.657921867" Jul 14 19:21:17.309664 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:17.309623 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9jbtj" podStartSLOduration=1.66302113 podStartE2EDuration="5.309612647s" totalImagesPullingTime="3.646591517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:21:17.309312916 +0000 UTC m=+626.674383280" watchObservedRunningTime="2026-07-14 19:21:17.309612647 +0000 UTC m=+626.674683008" Jul 14 19:21:18.263567 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:18.263532 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:18.422443 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:18.422413 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:18.878367 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:18.878332 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:19.164620 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:19.164539 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:19.341803 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:19.341775 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:19.872075 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:19.872019 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk"] Jul 14 19:21:19.875238 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:19.875222 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:19.878908 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:19.878876 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 19:21:19.879085 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:19.878953 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 19:21:19.879085 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:19.878964 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 19:21:19.879085 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:19.878975 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:21:19.879085 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:19.878976 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-q676c\"" Jul 14 19:21:19.879085 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:19.878951 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 19:21:19.886848 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:19.886826 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk"] Jul 14 19:21:20.036890 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:20.036853 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cfa40f7-76f3-4f0b-ba97-3c2b96943e58-cert\") pod \"lws-controller-manager-59cf65bbd6-ttggk\" (UID: \"2cfa40f7-76f3-4f0b-ba97-3c2b96943e58\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:20.037083 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:20.036917 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cfa40f7-76f3-4f0b-ba97-3c2b96943e58-metrics-cert\") pod \"lws-controller-manager-59cf65bbd6-ttggk\" (UID: \"2cfa40f7-76f3-4f0b-ba97-3c2b96943e58\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:20.037083 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:20.036956 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fvxb\" (UniqueName: \"kubernetes.io/projected/2cfa40f7-76f3-4f0b-ba97-3c2b96943e58-kube-api-access-5fvxb\") pod \"lws-controller-manager-59cf65bbd6-ttggk\" (UID: \"2cfa40f7-76f3-4f0b-ba97-3c2b96943e58\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:20.037083 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:20.036999 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2cfa40f7-76f3-4f0b-ba97-3c2b96943e58-manager-config\") pod \"lws-controller-manager-59cf65bbd6-ttggk\" (UID: \"2cfa40f7-76f3-4f0b-ba97-3c2b96943e58\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:20.138270 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:20.138189 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cfa40f7-76f3-4f0b-ba97-3c2b96943e58-cert\") pod \"lws-controller-manager-59cf65bbd6-ttggk\" (UID: \"2cfa40f7-76f3-4f0b-ba97-3c2b96943e58\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:20.138270 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:20.138230 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cfa40f7-76f3-4f0b-ba97-3c2b96943e58-metrics-cert\") pod \"lws-controller-manager-59cf65bbd6-ttggk\" (UID: \"2cfa40f7-76f3-4f0b-ba97-3c2b96943e58\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:20.138270 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:20.138256 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fvxb\" (UniqueName: \"kubernetes.io/projected/2cfa40f7-76f3-4f0b-ba97-3c2b96943e58-kube-api-access-5fvxb\") pod \"lws-controller-manager-59cf65bbd6-ttggk\" (UID: \"2cfa40f7-76f3-4f0b-ba97-3c2b96943e58\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:20.138270 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:20.138275 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2cfa40f7-76f3-4f0b-ba97-3c2b96943e58-manager-config\") pod \"lws-controller-manager-59cf65bbd6-ttggk\" (UID: \"2cfa40f7-76f3-4f0b-ba97-3c2b96943e58\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:20.138837 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:20.138816 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2cfa40f7-76f3-4f0b-ba97-3c2b96943e58-manager-config\") pod \"lws-controller-manager-59cf65bbd6-ttggk\" (UID: \"2cfa40f7-76f3-4f0b-ba97-3c2b96943e58\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:20.140798 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:20.140766 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cfa40f7-76f3-4f0b-ba97-3c2b96943e58-cert\") pod \"lws-controller-manager-59cf65bbd6-ttggk\" (UID: \"2cfa40f7-76f3-4f0b-ba97-3c2b96943e58\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:20.140904 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:20.140812 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cfa40f7-76f3-4f0b-ba97-3c2b96943e58-metrics-cert\") pod \"lws-controller-manager-59cf65bbd6-ttggk\" (UID: \"2cfa40f7-76f3-4f0b-ba97-3c2b96943e58\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:20.150442 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:20.150415 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fvxb\" (UniqueName: \"kubernetes.io/projected/2cfa40f7-76f3-4f0b-ba97-3c2b96943e58-kube-api-access-5fvxb\") pod \"lws-controller-manager-59cf65bbd6-ttggk\" (UID: \"2cfa40f7-76f3-4f0b-ba97-3c2b96943e58\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:20.184668 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:20.184629 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:20.329074 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:20.329035 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk"] Jul 14 19:21:20.330575 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:21:20.330556 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cfa40f7_76f3_4f0b_ba97_3c2b96943e58.slice/crio-5b6a6c44be6fab1302227b4d8a4805b1cadeba963b67b1c31acee2c839acb7f8 WatchSource:0}: Error finding container 5b6a6c44be6fab1302227b4d8a4805b1cadeba963b67b1c31acee2c839acb7f8: Status 404 returned error can't find the container with id 5b6a6c44be6fab1302227b4d8a4805b1cadeba963b67b1c31acee2c839acb7f8 Jul 14 19:21:21.295020 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:21.294977 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" event={"ID":"2cfa40f7-76f3-4f0b-ba97-3c2b96943e58","Type":"ContainerStarted","Data":"5b6a6c44be6fab1302227b4d8a4805b1cadeba963b67b1c31acee2c839acb7f8"} Jul 14 19:21:23.304174 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:23.304133 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" event={"ID":"2cfa40f7-76f3-4f0b-ba97-3c2b96943e58","Type":"ContainerStarted","Data":"a176f4745e505d01b5761ec86fa6c6ef3f97eb3ccfb2a1326b78908a28d1f626"} Jul 14 19:21:23.304579 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:23.304373 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:21:23.341836 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:23.341787 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" podStartSLOduration=1.934606006 podStartE2EDuration="4.341768713s" totalImagesPullingTime="2.407162707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:21:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:21:23.339686295 +0000 UTC m=+632.704756656" watchObservedRunningTime="2026-07-14 19:21:23.341768713 +0000 UTC m=+632.706839075" Jul 14 19:21:23.528517 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:23.528480 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:23.724804 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:23.724766 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:23.872520 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:23.872488 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:23.986697 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:23.986608 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:24.122822 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:24.122790 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:21:28.283576 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:28.283543 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9jbtj" Jul 14 19:21:34.310901 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:21:34.310870 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-ttggk" Jul 14 19:22:13.030068 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:13.030024 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:13.586645 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:13.586610 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:13.716465 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:13.716435 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:13.853659 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:13.853565 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:14.537404 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:14.537371 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:14.744389 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:14.744348 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:17.537420 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:17.537382 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:17.810118 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:17.810005 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:28.316446 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:28.316409 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:28.428280 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:28.428246 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:44.016686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:44.016582 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:47.638199 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:47.638150 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:49.444444 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:49.444407 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:51.002357 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:51.002321 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:51.398413 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:51.398369 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:54.731864 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:54.731821 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:54.834910 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:54.834875 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:55.248192 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:55.248158 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:55.350563 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:55.350518 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:57.239118 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:57.239071 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:58.223444 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:58.223408 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:58.354468 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:58.354435 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:59.004387 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:59.004354 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:22:59.065284 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:22:59.065252 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:06.665223 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:06.665186 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:06.801053 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:06.801006 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:11.422558 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:11.422520 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:11.523537 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:11.523499 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:13.853750 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:13.853715 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:13.912346 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:13.912307 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:16.537895 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:16.537856 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:16.638758 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:16.638724 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:19.432220 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:19.432183 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:19.530241 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:19.530202 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:26.285784 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:26.285747 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:26.386187 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:26.386148 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:30.111024 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:30.110990 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:31.092710 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:31.092678 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:31.185977 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:31.185939 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:33.527561 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:33.527532 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:33.627587 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:33.627547 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:39.481087 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:39.481021 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:39.585062 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:39.585006 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:46.723308 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:46.723275 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:46.830570 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:46.830531 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:50.888353 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:50.888319 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:51.248802 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:51.248714 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:51.606401 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:51.606355 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:51.708288 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:51.708243 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:51.835508 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:51.835476 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:53.958773 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:53.958728 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:23:54.054493 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:23:54.054452 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:24:17.719251 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:17.719167 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:24:17.877554 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:17.877514 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:24:17.968102 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:17.968069 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:24:18.208578 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.208550 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:24:18.211799 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.211783 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-pflx5" Jul 14 19:24:18.214455 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.214428 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 19:24:18.215290 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.215275 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 19:24:18.215388 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.215352 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 14 19:24:18.215388 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.215364 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jwxt4\"" Jul 14 19:24:18.220282 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.220262 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:24:18.251537 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.251512 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:24:18.285279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.285252 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4993e8f1-f4c4-43f2-910f-92cc3f74c2af-config-file\") pod \"limitador-limitador-54d6d988d5-pflx5\" (UID: \"4993e8f1-f4c4-43f2-910f-92cc3f74c2af\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pflx5" Jul 14 19:24:18.285415 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.285327 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq6lv\" (UniqueName: \"kubernetes.io/projected/4993e8f1-f4c4-43f2-910f-92cc3f74c2af-kube-api-access-pq6lv\") pod \"limitador-limitador-54d6d988d5-pflx5\" (UID: \"4993e8f1-f4c4-43f2-910f-92cc3f74c2af\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pflx5" Jul 14 19:24:18.386272 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.386233 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4993e8f1-f4c4-43f2-910f-92cc3f74c2af-config-file\") pod \"limitador-limitador-54d6d988d5-pflx5\" (UID: \"4993e8f1-f4c4-43f2-910f-92cc3f74c2af\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pflx5" Jul 14 19:24:18.386450 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.386280 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq6lv\" (UniqueName: \"kubernetes.io/projected/4993e8f1-f4c4-43f2-910f-92cc3f74c2af-kube-api-access-pq6lv\") pod \"limitador-limitador-54d6d988d5-pflx5\" (UID: \"4993e8f1-f4c4-43f2-910f-92cc3f74c2af\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pflx5" Jul 14 19:24:18.386953 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.386933 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4993e8f1-f4c4-43f2-910f-92cc3f74c2af-config-file\") pod \"limitador-limitador-54d6d988d5-pflx5\" (UID: \"4993e8f1-f4c4-43f2-910f-92cc3f74c2af\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pflx5" Jul 14 19:24:18.394720 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.394699 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq6lv\" (UniqueName: \"kubernetes.io/projected/4993e8f1-f4c4-43f2-910f-92cc3f74c2af-kube-api-access-pq6lv\") pod \"limitador-limitador-54d6d988d5-pflx5\" (UID: \"4993e8f1-f4c4-43f2-910f-92cc3f74c2af\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pflx5" Jul 14 19:24:18.523262 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.523172 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-pflx5" Jul 14 19:24:18.569132 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.569096 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-hf9gq"] Jul 14 19:24:18.572770 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.572745 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-hf9gq" Jul 14 19:24:18.575917 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.575683 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qdr5n\"" Jul 14 19:24:18.580155 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.580113 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-hf9gq"] Jul 14 19:24:18.587354 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.587323 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsk4k\" (UniqueName: \"kubernetes.io/projected/0588be5c-81f2-4448-a3ef-9585b4ca7096-kube-api-access-lsk4k\") pod \"authorino-58cd955675-hf9gq\" (UID: \"0588be5c-81f2-4448-a3ef-9585b4ca7096\") " pod="kuadrant-system/authorino-58cd955675-hf9gq" Jul 14 19:24:18.602379 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.602207 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:24:18.658415 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.658378 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:24:18.667094 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.667068 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:24:18.669548 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:24:18.669521 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4993e8f1_f4c4_43f2_910f_92cc3f74c2af.slice/crio-929219acaa9d0d48b0a691387d9d48666eb1ccd3925362bfd341217e5158985f WatchSource:0}: Error finding container 929219acaa9d0d48b0a691387d9d48666eb1ccd3925362bfd341217e5158985f: Status 404 returned error can't find the container with id 929219acaa9d0d48b0a691387d9d48666eb1ccd3925362bfd341217e5158985f Jul 14 19:24:18.671466 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.671446 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:24:18.688333 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.688303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsk4k\" (UniqueName: \"kubernetes.io/projected/0588be5c-81f2-4448-a3ef-9585b4ca7096-kube-api-access-lsk4k\") pod \"authorino-58cd955675-hf9gq\" (UID: \"0588be5c-81f2-4448-a3ef-9585b4ca7096\") " pod="kuadrant-system/authorino-58cd955675-hf9gq" Jul 14 19:24:18.697246 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.697213 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsk4k\" (UniqueName: \"kubernetes.io/projected/0588be5c-81f2-4448-a3ef-9585b4ca7096-kube-api-access-lsk4k\") pod \"authorino-58cd955675-hf9gq\" (UID: \"0588be5c-81f2-4448-a3ef-9585b4ca7096\") " pod="kuadrant-system/authorino-58cd955675-hf9gq" Jul 14 19:24:18.886660 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.886625 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-hf9gq" Jul 14 19:24:18.940314 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:18.940279 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-pflx5" event={"ID":"4993e8f1-f4c4-43f2-910f-92cc3f74c2af","Type":"ContainerStarted","Data":"929219acaa9d0d48b0a691387d9d48666eb1ccd3925362bfd341217e5158985f"} Jul 14 19:24:19.017257 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:19.017221 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-hf9gq"] Jul 14 19:24:19.019961 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:24:19.019932 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0588be5c_81f2_4448_a3ef_9585b4ca7096.slice/crio-2a3cadba0e2d181b6457b98d780a3a5c6075bab73294782dc61b1ed7fd106398 WatchSource:0}: Error finding container 2a3cadba0e2d181b6457b98d780a3a5c6075bab73294782dc61b1ed7fd106398: Status 404 returned error can't find the container with id 2a3cadba0e2d181b6457b98d780a3a5c6075bab73294782dc61b1ed7fd106398 Jul 14 19:24:19.947521 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:19.947473 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-hf9gq" event={"ID":"0588be5c-81f2-4448-a3ef-9585b4ca7096","Type":"ContainerStarted","Data":"2a3cadba0e2d181b6457b98d780a3a5c6075bab73294782dc61b1ed7fd106398"} Jul 14 19:24:23.709571 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:23.709532 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-hf9gq"] Jul 14 19:24:23.969117 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:23.968995 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-pflx5" event={"ID":"4993e8f1-f4c4-43f2-910f-92cc3f74c2af","Type":"ContainerStarted","Data":"68618ef78b8a5ab957b61a0959aeefa5f377467ba4fb8dfa6e386a48c7704d99"} Jul 14 19:24:23.969271 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:23.969193 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-pflx5" Jul 14 19:24:23.970428 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:23.970405 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-hf9gq" event={"ID":"0588be5c-81f2-4448-a3ef-9585b4ca7096","Type":"ContainerStarted","Data":"8e87296f46b271455a66edb291f8a56368883b2fcdba2913881bdff9acd80313"} Jul 14 19:24:23.985791 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:23.985734 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-pflx5" podStartSLOduration=1.656150061 podStartE2EDuration="5.985717184s" totalImagesPullingTime="4.329567123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:24:23.985115015 +0000 UTC m=+813.350185373" watchObservedRunningTime="2026-07-14 19:24:23.985717184 +0000 UTC m=+813.350787545" Jul 14 19:24:24.001164 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:24.001106 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-hf9gq" podStartSLOduration=2.059307919 podStartE2EDuration="6.001090378s" totalImagesPullingTime="3.941782459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:24:23.999726963 +0000 UTC m=+813.364797321" watchObservedRunningTime="2026-07-14 19:24:24.001090378 +0000 UTC m=+813.366160749" Jul 14 19:24:24.974518 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:24.974473 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-hf9gq" podUID="0588be5c-81f2-4448-a3ef-9585b4ca7096" containerName="authorino" containerID="cri-o://8e87296f46b271455a66edb291f8a56368883b2fcdba2913881bdff9acd80313" gracePeriod=30 Jul 14 19:24:25.222191 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:25.222167 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-hf9gq" Jul 14 19:24:25.246588 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:25.246505 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsk4k\" (UniqueName: \"kubernetes.io/projected/0588be5c-81f2-4448-a3ef-9585b4ca7096-kube-api-access-lsk4k\") pod \"0588be5c-81f2-4448-a3ef-9585b4ca7096\" (UID: \"0588be5c-81f2-4448-a3ef-9585b4ca7096\") " Jul 14 19:24:25.248610 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:25.248579 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0588be5c-81f2-4448-a3ef-9585b4ca7096-kube-api-access-lsk4k" (OuterVolumeSpecName: "kube-api-access-lsk4k") pod "0588be5c-81f2-4448-a3ef-9585b4ca7096" (UID: "0588be5c-81f2-4448-a3ef-9585b4ca7096"). InnerVolumeSpecName "kube-api-access-lsk4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:24:25.347874 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:25.347826 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsk4k\" (UniqueName: \"kubernetes.io/projected/0588be5c-81f2-4448-a3ef-9585b4ca7096-kube-api-access-lsk4k\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:24:25.979616 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:25.979581 2572 generic.go:356] "Generic (PLEG): container finished" podID="0588be5c-81f2-4448-a3ef-9585b4ca7096" containerID="8e87296f46b271455a66edb291f8a56368883b2fcdba2913881bdff9acd80313" exitCode=0 Jul 14 19:24:25.980115 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:25.979640 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-hf9gq" Jul 14 19:24:25.980115 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:25.979672 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-hf9gq" event={"ID":"0588be5c-81f2-4448-a3ef-9585b4ca7096","Type":"ContainerDied","Data":"8e87296f46b271455a66edb291f8a56368883b2fcdba2913881bdff9acd80313"} Jul 14 19:24:25.980115 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:25.979728 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-hf9gq" event={"ID":"0588be5c-81f2-4448-a3ef-9585b4ca7096","Type":"ContainerDied","Data":"2a3cadba0e2d181b6457b98d780a3a5c6075bab73294782dc61b1ed7fd106398"} Jul 14 19:24:25.980115 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:25.979756 2572 scope.go:117] "RemoveContainer" containerID="8e87296f46b271455a66edb291f8a56368883b2fcdba2913881bdff9acd80313" Jul 14 19:24:25.987681 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:25.987666 2572 scope.go:117] "RemoveContainer" containerID="8e87296f46b271455a66edb291f8a56368883b2fcdba2913881bdff9acd80313" Jul 14 19:24:25.987931 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:24:25.987914 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e87296f46b271455a66edb291f8a56368883b2fcdba2913881bdff9acd80313\": container with ID starting with 8e87296f46b271455a66edb291f8a56368883b2fcdba2913881bdff9acd80313 not found: ID does not exist" containerID="8e87296f46b271455a66edb291f8a56368883b2fcdba2913881bdff9acd80313" Jul 14 19:24:25.987982 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:25.987939 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e87296f46b271455a66edb291f8a56368883b2fcdba2913881bdff9acd80313"} err="failed to get container status \"8e87296f46b271455a66edb291f8a56368883b2fcdba2913881bdff9acd80313\": rpc error: code = NotFound desc = could not find container \"8e87296f46b271455a66edb291f8a56368883b2fcdba2913881bdff9acd80313\": container with ID starting with 8e87296f46b271455a66edb291f8a56368883b2fcdba2913881bdff9acd80313 not found: ID does not exist" Jul 14 19:24:26.001897 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:26.001869 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-hf9gq"] Jul 14 19:24:26.007443 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:26.007418 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-hf9gq"] Jul 14 19:24:27.187711 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:27.187680 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0588be5c-81f2-4448-a3ef-9585b4ca7096" path="/var/lib/kubelet/pods/0588be5c-81f2-4448-a3ef-9585b4ca7096/volumes" Jul 14 19:24:34.975870 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:34.975840 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-pflx5" Jul 14 19:24:38.940486 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:38.940454 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:24:39.042147 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:39.042105 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:24:39.435175 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:39.435127 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:24:51.118029 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.117989 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:24:51.218235 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.218204 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:24:51.347935 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.347903 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dlbsg"] Jul 14 19:24:51.348278 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.348260 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0588be5c-81f2-4448-a3ef-9585b4ca7096" containerName="authorino" Jul 14 19:24:51.348361 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.348281 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0588be5c-81f2-4448-a3ef-9585b4ca7096" containerName="authorino" Jul 14 19:24:51.348416 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.348363 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0588be5c-81f2-4448-a3ef-9585b4ca7096" containerName="authorino" Jul 14 19:24:51.352456 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.352436 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dlbsg" Jul 14 19:24:51.354553 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.354533 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qdr5n\"" Jul 14 19:24:51.357410 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.357385 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dlbsg"] Jul 14 19:24:51.448615 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.448574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvb7x\" (UniqueName: \"kubernetes.io/projected/eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1-kube-api-access-cvb7x\") pod \"authorino-5d4b67c446-dlbsg\" (UID: \"eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1\") " pod="kuadrant-system/authorino-5d4b67c446-dlbsg" Jul 14 19:24:51.525854 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.525820 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dlbsg"] Jul 14 19:24:51.526090 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:24:51.526069 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cvb7x], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-dlbsg" podUID="eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1" Jul 14 19:24:51.549600 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.549568 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvb7x\" (UniqueName: \"kubernetes.io/projected/eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1-kube-api-access-cvb7x\") pod \"authorino-5d4b67c446-dlbsg\" (UID: \"eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1\") " pod="kuadrant-system/authorino-5d4b67c446-dlbsg" Jul 14 19:24:51.557216 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.557188 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-9nh4j"] Jul 14 19:24:51.560224 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.560210 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-9nh4j" Jul 14 19:24:51.562765 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.562750 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 14 19:24:51.565536 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.565516 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvb7x\" (UniqueName: \"kubernetes.io/projected/eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1-kube-api-access-cvb7x\") pod \"authorino-5d4b67c446-dlbsg\" (UID: \"eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1\") " pod="kuadrant-system/authorino-5d4b67c446-dlbsg" Jul 14 19:24:51.570307 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.570290 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-9nh4j"] Jul 14 19:24:51.609927 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.609850 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-9nh4j"] Jul 14 19:24:51.610087 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:24:51.610071 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-8nkx7 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-9nh4j" podUID="9913c450-3ebd-4a70-9566-89f86cded23b" Jul 14 19:24:51.639686 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.639654 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7599fb7c89-7fn2z"] Jul 14 19:24:51.642821 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.642802 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7599fb7c89-7fn2z" Jul 14 19:24:51.650777 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.650750 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9913c450-3ebd-4a70-9566-89f86cded23b-tls-cert\") pod \"authorino-76877994-9nh4j\" (UID: \"9913c450-3ebd-4a70-9566-89f86cded23b\") " pod="kuadrant-system/authorino-76877994-9nh4j" Jul 14 19:24:51.650875 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.650792 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nkx7\" (UniqueName: \"kubernetes.io/projected/9913c450-3ebd-4a70-9566-89f86cded23b-kube-api-access-8nkx7\") pod \"authorino-76877994-9nh4j\" (UID: \"9913c450-3ebd-4a70-9566-89f86cded23b\") " pod="kuadrant-system/authorino-76877994-9nh4j" Jul 14 19:24:51.650978 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.650958 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7599fb7c89-7fn2z"] Jul 14 19:24:51.751420 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.751377 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3acce073-69a8-4e39-8528-5634d3459b1a-tls-cert\") pod \"authorino-7599fb7c89-7fn2z\" (UID: \"3acce073-69a8-4e39-8528-5634d3459b1a\") " pod="kuadrant-system/authorino-7599fb7c89-7fn2z" Jul 14 19:24:51.751596 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.751430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqnnh\" (UniqueName: \"kubernetes.io/projected/3acce073-69a8-4e39-8528-5634d3459b1a-kube-api-access-qqnnh\") pod \"authorino-7599fb7c89-7fn2z\" (UID: \"3acce073-69a8-4e39-8528-5634d3459b1a\") " pod="kuadrant-system/authorino-7599fb7c89-7fn2z" Jul 14 19:24:51.751596 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.751527 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9913c450-3ebd-4a70-9566-89f86cded23b-tls-cert\") pod \"authorino-76877994-9nh4j\" (UID: \"9913c450-3ebd-4a70-9566-89f86cded23b\") " pod="kuadrant-system/authorino-76877994-9nh4j" Jul 14 19:24:51.751596 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.751563 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nkx7\" (UniqueName: \"kubernetes.io/projected/9913c450-3ebd-4a70-9566-89f86cded23b-kube-api-access-8nkx7\") pod \"authorino-76877994-9nh4j\" (UID: \"9913c450-3ebd-4a70-9566-89f86cded23b\") " pod="kuadrant-system/authorino-76877994-9nh4j" Jul 14 19:24:51.754119 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.754082 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9913c450-3ebd-4a70-9566-89f86cded23b-tls-cert\") pod \"authorino-76877994-9nh4j\" (UID: \"9913c450-3ebd-4a70-9566-89f86cded23b\") " pod="kuadrant-system/authorino-76877994-9nh4j" Jul 14 19:24:51.761391 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.761359 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nkx7\" (UniqueName: \"kubernetes.io/projected/9913c450-3ebd-4a70-9566-89f86cded23b-kube-api-access-8nkx7\") pod \"authorino-76877994-9nh4j\" (UID: \"9913c450-3ebd-4a70-9566-89f86cded23b\") " pod="kuadrant-system/authorino-76877994-9nh4j" Jul 14 19:24:51.852617 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.852582 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3acce073-69a8-4e39-8528-5634d3459b1a-tls-cert\") pod \"authorino-7599fb7c89-7fn2z\" (UID: \"3acce073-69a8-4e39-8528-5634d3459b1a\") " pod="kuadrant-system/authorino-7599fb7c89-7fn2z" Jul 14 19:24:51.852617 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.852621 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqnnh\" (UniqueName: \"kubernetes.io/projected/3acce073-69a8-4e39-8528-5634d3459b1a-kube-api-access-qqnnh\") pod \"authorino-7599fb7c89-7fn2z\" (UID: \"3acce073-69a8-4e39-8528-5634d3459b1a\") " pod="kuadrant-system/authorino-7599fb7c89-7fn2z" Jul 14 19:24:51.855028 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.855005 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3acce073-69a8-4e39-8528-5634d3459b1a-tls-cert\") pod \"authorino-7599fb7c89-7fn2z\" (UID: \"3acce073-69a8-4e39-8528-5634d3459b1a\") " pod="kuadrant-system/authorino-7599fb7c89-7fn2z" Jul 14 19:24:51.861221 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.861163 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqnnh\" (UniqueName: \"kubernetes.io/projected/3acce073-69a8-4e39-8528-5634d3459b1a-kube-api-access-qqnnh\") pod \"authorino-7599fb7c89-7fn2z\" (UID: \"3acce073-69a8-4e39-8528-5634d3459b1a\") " pod="kuadrant-system/authorino-7599fb7c89-7fn2z" Jul 14 19:24:51.952130 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:51.952096 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7599fb7c89-7fn2z" Jul 14 19:24:52.078529 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:52.078495 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7599fb7c89-7fn2z"] Jul 14 19:24:52.081711 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:52.081685 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-9nh4j" Jul 14 19:24:52.081825 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:52.081717 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dlbsg" Jul 14 19:24:52.081938 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:24:52.081912 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3acce073_69a8_4e39_8528_5634d3459b1a.slice/crio-298f252eaa2b4c68cd8b8e1821cdff4e148d33e6b6746d5996587494f7aa2e17 WatchSource:0}: Error finding container 298f252eaa2b4c68cd8b8e1821cdff4e148d33e6b6746d5996587494f7aa2e17: Status 404 returned error can't find the container with id 298f252eaa2b4c68cd8b8e1821cdff4e148d33e6b6746d5996587494f7aa2e17 Jul 14 19:24:52.090431 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:52.090411 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-9nh4j" Jul 14 19:24:52.092560 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:52.092544 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dlbsg" Jul 14 19:24:52.154835 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:52.154746 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9913c450-3ebd-4a70-9566-89f86cded23b-tls-cert\") pod \"9913c450-3ebd-4a70-9566-89f86cded23b\" (UID: \"9913c450-3ebd-4a70-9566-89f86cded23b\") " Jul 14 19:24:52.154835 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:52.154803 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvb7x\" (UniqueName: \"kubernetes.io/projected/eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1-kube-api-access-cvb7x\") pod \"eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1\" (UID: \"eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1\") " Jul 14 19:24:52.154835 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:52.154822 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nkx7\" (UniqueName: \"kubernetes.io/projected/9913c450-3ebd-4a70-9566-89f86cded23b-kube-api-access-8nkx7\") pod \"9913c450-3ebd-4a70-9566-89f86cded23b\" (UID: \"9913c450-3ebd-4a70-9566-89f86cded23b\") " Jul 14 19:24:52.156951 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:52.156922 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9913c450-3ebd-4a70-9566-89f86cded23b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9913c450-3ebd-4a70-9566-89f86cded23b" (UID: "9913c450-3ebd-4a70-9566-89f86cded23b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:24:52.157083 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:52.156949 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9913c450-3ebd-4a70-9566-89f86cded23b-kube-api-access-8nkx7" (OuterVolumeSpecName: "kube-api-access-8nkx7") pod "9913c450-3ebd-4a70-9566-89f86cded23b" (UID: "9913c450-3ebd-4a70-9566-89f86cded23b"). InnerVolumeSpecName "kube-api-access-8nkx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:24:52.157083 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:52.156933 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1-kube-api-access-cvb7x" (OuterVolumeSpecName: "kube-api-access-cvb7x") pod "eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1" (UID: "eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1"). InnerVolumeSpecName "kube-api-access-cvb7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:24:52.255601 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:52.255572 2572 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9913c450-3ebd-4a70-9566-89f86cded23b-tls-cert\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:24:52.255601 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:52.255600 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvb7x\" (UniqueName: \"kubernetes.io/projected/eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1-kube-api-access-cvb7x\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:24:52.255790 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:52.255614 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nkx7\" (UniqueName: \"kubernetes.io/projected/9913c450-3ebd-4a70-9566-89f86cded23b-kube-api-access-8nkx7\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:24:53.087127 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:53.087086 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7599fb7c89-7fn2z" event={"ID":"3acce073-69a8-4e39-8528-5634d3459b1a","Type":"ContainerStarted","Data":"16b58d24d25ca242694b9e6e0d6a98178e1a9501f3d02d440dbedd70ea6aee6c"} Jul 14 19:24:53.087278 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:53.087139 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7599fb7c89-7fn2z" event={"ID":"3acce073-69a8-4e39-8528-5634d3459b1a","Type":"ContainerStarted","Data":"298f252eaa2b4c68cd8b8e1821cdff4e148d33e6b6746d5996587494f7aa2e17"} Jul 14 19:24:53.087278 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:53.087201 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dlbsg" Jul 14 19:24:53.087388 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:53.087287 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-9nh4j" Jul 14 19:24:53.102742 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:53.102701 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7599fb7c89-7fn2z" podStartSLOduration=1.6685135199999999 podStartE2EDuration="2.102685529s" totalImagesPullingTime="434.172009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:24:53.10215234 +0000 UTC m=+842.467222698" watchObservedRunningTime="2026-07-14 19:24:53.102685529 +0000 UTC m=+842.467755906" Jul 14 19:24:53.124896 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:53.124872 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dlbsg"] Jul 14 19:24:53.132830 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:53.132808 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dlbsg"] Jul 14 19:24:53.160767 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:53.160736 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-9nh4j"] Jul 14 19:24:53.162322 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:53.162305 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-9nh4j"] Jul 14 19:24:53.188508 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:53.188482 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9913c450-3ebd-4a70-9566-89f86cded23b" path="/var/lib/kubelet/pods/9913c450-3ebd-4a70-9566-89f86cded23b/volumes" Jul 14 19:24:53.188749 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:53.188732 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1" path="/var/lib/kubelet/pods/eb880ae9-db9d-4c3d-aa78-5d4273ee9ef1/volumes" Jul 14 19:24:57.856890 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:57.856855 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:24:57.958783 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:57.958753 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:24:58.049593 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:58.049554 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:24:58.183623 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:24:58.183533 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:25:01.050956 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:01.050924 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:25:01.138934 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:01.138896 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:25:31.851104 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:31.851068 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:25:31.951455 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:31.951426 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:25:32.978284 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:32.978251 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:25:34.551879 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:34.551844 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:25:34.652427 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:34.652392 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:25:37.582655 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:37.582622 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:25:37.681195 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:37.681162 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:25:44.532359 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:44.532276 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:25:44.648228 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:44.648188 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:25:51.057917 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:51.057878 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:25:51.059976 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:51.059956 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:25:52.696459 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:52.696427 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:25:53.144569 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:53.144529 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:25:56.365196 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:56.365156 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:25:58.633919 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:58.633881 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:25:59.532016 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:25:59.531975 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:26:08.024100 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:08.024069 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:26:09.327407 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:09.327362 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:26:19.355448 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:19.355413 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:26:20.743101 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:20.742426 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:26:31.819148 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:31.819107 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p"] Jul 14 19:26:31.822297 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:31.822282 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:31.825727 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:31.825703 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 14 19:26:31.825727 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:31.825713 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 19:26:31.825942 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:31.825743 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 19:26:31.825942 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:31.825717 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cxkgd\"" Jul 14 19:26:31.831365 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:31.831345 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p"] Jul 14 19:26:31.862379 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:31.862333 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:26:31.919945 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:31.919910 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbtht\" (UniqueName: \"kubernetes.io/projected/34844f98-eada-430e-9065-27f1f550ea5a-kube-api-access-fbtht\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:31.920121 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:31.919961 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34844f98-eada-430e-9065-27f1f550ea5a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:31.920121 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:31.920026 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34844f98-eada-430e-9065-27f1f550ea5a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:31.920121 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:31.920087 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34844f98-eada-430e-9065-27f1f550ea5a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:31.920121 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:31.920107 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34844f98-eada-430e-9065-27f1f550ea5a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:31.920282 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:31.920127 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34844f98-eada-430e-9065-27f1f550ea5a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:32.020782 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.020750 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34844f98-eada-430e-9065-27f1f550ea5a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:32.020959 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.020789 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34844f98-eada-430e-9065-27f1f550ea5a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:32.020959 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.020815 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34844f98-eada-430e-9065-27f1f550ea5a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:32.020959 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.020866 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbtht\" (UniqueName: \"kubernetes.io/projected/34844f98-eada-430e-9065-27f1f550ea5a-kube-api-access-fbtht\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:32.020959 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.020905 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34844f98-eada-430e-9065-27f1f550ea5a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:32.020959 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.020933 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34844f98-eada-430e-9065-27f1f550ea5a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:32.021241 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.021192 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34844f98-eada-430e-9065-27f1f550ea5a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:32.021298 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.021236 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34844f98-eada-430e-9065-27f1f550ea5a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:32.021298 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.021260 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34844f98-eada-430e-9065-27f1f550ea5a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:32.023118 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.023093 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34844f98-eada-430e-9065-27f1f550ea5a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:32.023403 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.023385 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34844f98-eada-430e-9065-27f1f550ea5a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:32.028679 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.028655 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbtht\" (UniqueName: \"kubernetes.io/projected/34844f98-eada-430e-9065-27f1f550ea5a-kube-api-access-fbtht\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-njn5p\" (UID: \"34844f98-eada-430e-9065-27f1f550ea5a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:32.133548 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.133517 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:32.258136 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.257972 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p"] Jul 14 19:26:32.260414 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:26:32.260383 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34844f98_eada_430e_9065_27f1f550ea5a.slice/crio-96657df97d7eff24438652505407f2ee7873826ad9e9117119469861ef929cb0 WatchSource:0}: Error finding container 96657df97d7eff24438652505407f2ee7873826ad9e9117119469861ef929cb0: Status 404 returned error can't find the container with id 96657df97d7eff24438652505407f2ee7873826ad9e9117119469861ef929cb0 Jul 14 19:26:32.476504 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:32.476428 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" event={"ID":"34844f98-eada-430e-9065-27f1f550ea5a","Type":"ContainerStarted","Data":"96657df97d7eff24438652505407f2ee7873826ad9e9117119469861ef929cb0"} Jul 14 19:26:33.484356 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:33.484325 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" event={"ID":"34844f98-eada-430e-9065-27f1f550ea5a","Type":"ContainerStarted","Data":"5d405a65459704bb8af88e8246ca6f5742fc51146724f9431448fae6f3d95dd8"} Jul 14 19:26:33.503731 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:33.503680 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" podStartSLOduration=1.4869015540000001 podStartE2EDuration="2.503664569s" totalImagesPullingTime="1.016763015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:26:33.502343685 +0000 UTC m=+942.867414046" watchObservedRunningTime="2026-07-14 19:26:33.503664569 +0000 UTC m=+942.868734930" Jul 14 19:26:33.835152 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:33.835113 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:26:42.134029 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:42.133995 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:42.134543 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:42.134097 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:42.146317 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:42.146294 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:42.539756 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:42.539681 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-njn5p" Jul 14 19:26:46.003771 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.003735 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b"] Jul 14 19:26:46.007132 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.007115 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.009649 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.009629 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 14 19:26:46.013813 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.013636 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55661263-8e16-43f8-9621-34185009df14-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.013813 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.013686 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55661263-8e16-43f8-9621-34185009df14-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.013813 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.013715 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftk9t\" (UniqueName: \"kubernetes.io/projected/55661263-8e16-43f8-9621-34185009df14-kube-api-access-ftk9t\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.013813 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.013749 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55661263-8e16-43f8-9621-34185009df14-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.013991 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.013860 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55661263-8e16-43f8-9621-34185009df14-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.013991 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.013896 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55661263-8e16-43f8-9621-34185009df14-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.017300 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.017278 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b"] Jul 14 19:26:46.114227 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.114188 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55661263-8e16-43f8-9621-34185009df14-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.114401 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.114234 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55661263-8e16-43f8-9621-34185009df14-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.114401 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.114260 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55661263-8e16-43f8-9621-34185009df14-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.114401 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.114285 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55661263-8e16-43f8-9621-34185009df14-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.114401 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.114301 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftk9t\" (UniqueName: \"kubernetes.io/projected/55661263-8e16-43f8-9621-34185009df14-kube-api-access-ftk9t\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.114401 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.114325 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55661263-8e16-43f8-9621-34185009df14-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.114699 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.114674 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55661263-8e16-43f8-9621-34185009df14-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.114772 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.114683 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55661263-8e16-43f8-9621-34185009df14-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.114772 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.114737 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55661263-8e16-43f8-9621-34185009df14-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.116661 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.116634 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55661263-8e16-43f8-9621-34185009df14-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.116795 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.116779 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55661263-8e16-43f8-9621-34185009df14-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.122490 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.122455 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftk9t\" (UniqueName: \"kubernetes.io/projected/55661263-8e16-43f8-9621-34185009df14-kube-api-access-ftk9t\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b\" (UID: \"55661263-8e16-43f8-9621-34185009df14\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.126531 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.126505 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:26:46.316887 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.316804 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:46.447647 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.447622 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b"] Jul 14 19:26:46.450267 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:26:46.450237 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55661263_8e16_43f8_9621_34185009df14.slice/crio-5a52d633881a6ed901c5f0db71769401e1df49a942668c6c6c0b24aacdba1123 WatchSource:0}: Error finding container 5a52d633881a6ed901c5f0db71769401e1df49a942668c6c6c0b24aacdba1123: Status 404 returned error can't find the container with id 5a52d633881a6ed901c5f0db71769401e1df49a942668c6c6c0b24aacdba1123 Jul 14 19:26:46.543213 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:46.543179 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" event={"ID":"55661263-8e16-43f8-9621-34185009df14","Type":"ContainerStarted","Data":"5a52d633881a6ed901c5f0db71769401e1df49a942668c6c6c0b24aacdba1123"} Jul 14 19:26:47.548367 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:47.548325 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" event={"ID":"55661263-8e16-43f8-9621-34185009df14","Type":"ContainerStarted","Data":"1fd312fda9b3c36696414aa95d64f0d6135e007b8055900d0dbc258b3e571102"} Jul 14 19:26:47.565645 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:47.565596 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" podStartSLOduration=2.387363402 podStartE2EDuration="2.565581435s" totalImagesPullingTime="178.218033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:26:47.564976702 +0000 UTC m=+956.930047064" watchObservedRunningTime="2026-07-14 19:26:47.565581435 +0000 UTC m=+956.930651796" Jul 14 19:26:48.633398 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:48.633361 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:26:56.317028 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:56.316983 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:56.317450 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:56.317060 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:56.329333 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:56.329304 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:26:56.595887 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:26:56.595856 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b" Jul 14 19:27:39.210681 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:39.210644 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:27:43.093152 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:43.093119 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6479548764-4w7zb"] Jul 14 19:27:43.096356 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:43.096336 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6479548764-4w7zb" Jul 14 19:27:43.103913 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:43.103885 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6479548764-4w7zb"] Jul 14 19:27:43.282663 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:43.282624 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ljpk\" (UniqueName: \"kubernetes.io/projected/2268ec04-5a0a-482c-9b77-35a7fcc01f0c-kube-api-access-6ljpk\") pod \"authorino-6479548764-4w7zb\" (UID: \"2268ec04-5a0a-482c-9b77-35a7fcc01f0c\") " pod="kuadrant-system/authorino-6479548764-4w7zb" Jul 14 19:27:43.282844 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:43.282689 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2268ec04-5a0a-482c-9b77-35a7fcc01f0c-tls-cert\") pod \"authorino-6479548764-4w7zb\" (UID: \"2268ec04-5a0a-482c-9b77-35a7fcc01f0c\") " pod="kuadrant-system/authorino-6479548764-4w7zb" Jul 14 19:27:43.383981 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:43.383893 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2268ec04-5a0a-482c-9b77-35a7fcc01f0c-tls-cert\") pod \"authorino-6479548764-4w7zb\" (UID: \"2268ec04-5a0a-482c-9b77-35a7fcc01f0c\") " pod="kuadrant-system/authorino-6479548764-4w7zb" Jul 14 19:27:43.383981 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:43.383969 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ljpk\" (UniqueName: \"kubernetes.io/projected/2268ec04-5a0a-482c-9b77-35a7fcc01f0c-kube-api-access-6ljpk\") pod \"authorino-6479548764-4w7zb\" (UID: \"2268ec04-5a0a-482c-9b77-35a7fcc01f0c\") " pod="kuadrant-system/authorino-6479548764-4w7zb" Jul 14 19:27:43.386294 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:43.386273 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2268ec04-5a0a-482c-9b77-35a7fcc01f0c-tls-cert\") pod \"authorino-6479548764-4w7zb\" (UID: \"2268ec04-5a0a-482c-9b77-35a7fcc01f0c\") " pod="kuadrant-system/authorino-6479548764-4w7zb" Jul 14 19:27:43.391440 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:43.391414 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ljpk\" (UniqueName: \"kubernetes.io/projected/2268ec04-5a0a-482c-9b77-35a7fcc01f0c-kube-api-access-6ljpk\") pod \"authorino-6479548764-4w7zb\" (UID: \"2268ec04-5a0a-482c-9b77-35a7fcc01f0c\") " pod="kuadrant-system/authorino-6479548764-4w7zb" Jul 14 19:27:43.406243 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:43.406211 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6479548764-4w7zb" Jul 14 19:27:43.541392 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:43.541367 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6479548764-4w7zb"] Jul 14 19:27:43.543448 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:27:43.543407 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2268ec04_5a0a_482c_9b77_35a7fcc01f0c.slice/crio-3dd4d09a4fd51350e0460661938392daa36ae0298ff6878db40236209f4d0d2d WatchSource:0}: Error finding container 3dd4d09a4fd51350e0460661938392daa36ae0298ff6878db40236209f4d0d2d: Status 404 returned error can't find the container with id 3dd4d09a4fd51350e0460661938392daa36ae0298ff6878db40236209f4d0d2d Jul 14 19:27:43.777696 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:43.777608 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6479548764-4w7zb" event={"ID":"2268ec04-5a0a-482c-9b77-35a7fcc01f0c","Type":"ContainerStarted","Data":"3dd4d09a4fd51350e0460661938392daa36ae0298ff6878db40236209f4d0d2d"} Jul 14 19:27:44.790211 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:44.790165 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6479548764-4w7zb" event={"ID":"2268ec04-5a0a-482c-9b77-35a7fcc01f0c","Type":"ContainerStarted","Data":"39828ce560a17482b980017ac99d4e8f6aed575dd5773eb7de7655c22913676e"} Jul 14 19:27:44.805681 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:44.805616 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6479548764-4w7zb" podStartSLOduration=1.392874559 podStartE2EDuration="1.805594744s" totalImagesPullingTime="412.720185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:27:44.804619811 +0000 UTC m=+1014.169690173" watchObservedRunningTime="2026-07-14 19:27:44.805594744 +0000 UTC m=+1014.170665106" Jul 14 19:27:44.836900 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:44.836863 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7599fb7c89-7fn2z"] Jul 14 19:27:44.837134 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:44.837108 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7599fb7c89-7fn2z" podUID="3acce073-69a8-4e39-8528-5634d3459b1a" containerName="authorino" containerID="cri-o://16b58d24d25ca242694b9e6e0d6a98178e1a9501f3d02d440dbedd70ea6aee6c" gracePeriod=30 Jul 14 19:27:45.102697 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.102673 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7599fb7c89-7fn2z" Jul 14 19:27:45.199188 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.199155 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3acce073-69a8-4e39-8528-5634d3459b1a-tls-cert\") pod \"3acce073-69a8-4e39-8528-5634d3459b1a\" (UID: \"3acce073-69a8-4e39-8528-5634d3459b1a\") " Jul 14 19:27:45.199340 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.199221 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqnnh\" (UniqueName: \"kubernetes.io/projected/3acce073-69a8-4e39-8528-5634d3459b1a-kube-api-access-qqnnh\") pod \"3acce073-69a8-4e39-8528-5634d3459b1a\" (UID: \"3acce073-69a8-4e39-8528-5634d3459b1a\") " Jul 14 19:27:45.201870 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.201832 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acce073-69a8-4e39-8528-5634d3459b1a-kube-api-access-qqnnh" (OuterVolumeSpecName: "kube-api-access-qqnnh") pod "3acce073-69a8-4e39-8528-5634d3459b1a" (UID: "3acce073-69a8-4e39-8528-5634d3459b1a"). InnerVolumeSpecName "kube-api-access-qqnnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:27:45.210314 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.210276 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3acce073-69a8-4e39-8528-5634d3459b1a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3acce073-69a8-4e39-8528-5634d3459b1a" (UID: "3acce073-69a8-4e39-8528-5634d3459b1a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:27:45.300175 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.300126 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqnnh\" (UniqueName: \"kubernetes.io/projected/3acce073-69a8-4e39-8528-5634d3459b1a-kube-api-access-qqnnh\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:27:45.300175 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.300162 2572 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3acce073-69a8-4e39-8528-5634d3459b1a-tls-cert\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:27:45.387564 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.387319 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:27:45.474502 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.474450 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:27:45.796530 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.796435 2572 generic.go:356] "Generic (PLEG): container finished" podID="3acce073-69a8-4e39-8528-5634d3459b1a" containerID="16b58d24d25ca242694b9e6e0d6a98178e1a9501f3d02d440dbedd70ea6aee6c" exitCode=0 Jul 14 19:27:45.796530 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.796498 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7599fb7c89-7fn2z" Jul 14 19:27:45.796994 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.796524 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7599fb7c89-7fn2z" event={"ID":"3acce073-69a8-4e39-8528-5634d3459b1a","Type":"ContainerDied","Data":"16b58d24d25ca242694b9e6e0d6a98178e1a9501f3d02d440dbedd70ea6aee6c"} Jul 14 19:27:45.796994 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.796568 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7599fb7c89-7fn2z" event={"ID":"3acce073-69a8-4e39-8528-5634d3459b1a","Type":"ContainerDied","Data":"298f252eaa2b4c68cd8b8e1821cdff4e148d33e6b6746d5996587494f7aa2e17"} Jul 14 19:27:45.796994 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.796587 2572 scope.go:117] "RemoveContainer" containerID="16b58d24d25ca242694b9e6e0d6a98178e1a9501f3d02d440dbedd70ea6aee6c" Jul 14 19:27:45.805012 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.804993 2572 scope.go:117] "RemoveContainer" containerID="16b58d24d25ca242694b9e6e0d6a98178e1a9501f3d02d440dbedd70ea6aee6c" Jul 14 19:27:45.805264 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:27:45.805244 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b58d24d25ca242694b9e6e0d6a98178e1a9501f3d02d440dbedd70ea6aee6c\": container with ID starting with 16b58d24d25ca242694b9e6e0d6a98178e1a9501f3d02d440dbedd70ea6aee6c not found: ID does not exist" containerID="16b58d24d25ca242694b9e6e0d6a98178e1a9501f3d02d440dbedd70ea6aee6c" Jul 14 19:27:45.805312 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.805273 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b58d24d25ca242694b9e6e0d6a98178e1a9501f3d02d440dbedd70ea6aee6c"} err="failed to get container status \"16b58d24d25ca242694b9e6e0d6a98178e1a9501f3d02d440dbedd70ea6aee6c\": rpc error: code = NotFound desc = could not find container \"16b58d24d25ca242694b9e6e0d6a98178e1a9501f3d02d440dbedd70ea6aee6c\": container with ID starting with 16b58d24d25ca242694b9e6e0d6a98178e1a9501f3d02d440dbedd70ea6aee6c not found: ID does not exist" Jul 14 19:27:45.821993 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.821965 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7599fb7c89-7fn2z"] Jul 14 19:27:45.835620 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:45.835595 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7599fb7c89-7fn2z"] Jul 14 19:27:46.559741 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:46.559711 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:27:46.665594 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:46.665561 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:27:47.188350 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:27:47.188318 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3acce073-69a8-4e39-8528-5634d3459b1a" path="/var/lib/kubelet/pods/3acce073-69a8-4e39-8528-5634d3459b1a/volumes" Jul 14 19:28:14.534658 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:28:14.534622 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:28:25.228005 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:28:25.227968 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:28:36.242290 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:28:36.242256 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:28:36.395805 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:28:36.395768 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:28:36.502867 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:28:36.502771 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:28:44.548672 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:28:44.548576 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:28:44.714440 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:28:44.714399 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:28:44.929967 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:28:44.929934 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:28:56.032959 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:28:56.032915 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:28:58.659266 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:28:58.659236 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:28:59.871192 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:28:59.871157 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:29:24.132784 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:29:24.132744 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:29:35.128014 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:29:35.127978 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:29:40.655159 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:29:40.655118 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:29:40.752207 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:29:40.752171 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:29:40.917244 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:29:40.917157 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:29:46.646928 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:29:46.646890 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:29:46.877591 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:29:46.877563 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:29:46.984561 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:29:46.984464 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:29:47.089685 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:29:47.089646 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:29:47.196606 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:29:47.196567 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:29:47.297652 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:29:47.297566 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:29:53.664130 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:29:53.664094 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:29:53.754329 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:29:53.754290 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:30:09.230608 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:30:09.230572 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:30:17.243170 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:30:17.243089 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:30:28.038406 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:30:28.038372 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:30:33.946964 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:30:33.946925 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:30:44.936477 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:30:44.936443 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:30:51.081075 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:30:51.081035 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:30:51.083896 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:30:51.083875 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:30:55.741464 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:30:55.741429 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:31:03.041769 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:31:03.041731 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:31:07.426283 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:31:07.426246 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:31:17.838793 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:31:17.838758 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:31:31.190505 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:31:31.190471 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:31:39.828402 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:31:39.828363 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:31:49.236972 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:31:49.236896 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:31:59.433110 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:31:59.433077 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:31:59.492660 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:31:59.492628 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:32:13.940215 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:32:13.940179 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:32:26.475267 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:32:26.475234 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:32:43.035596 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:32:43.035558 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:32:53.037287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:32:53.037249 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:33:02.953248 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:33:02.953215 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:33:12.282669 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:33:12.282589 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:33:12.441433 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:33:12.441402 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:33:22.056098 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:33:22.056060 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:35:11.734190 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:35:11.734107 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:35:23.434176 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:35:23.434142 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:35:24.592873 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:35:24.592839 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:35:32.536462 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:35:32.536424 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:35:37.818964 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:35:37.818936 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:35:41.731969 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:35:41.731930 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:35:50.936018 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:35:50.935979 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:35:50.943621 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:35:50.943583 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:35:51.103682 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:35:51.103646 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:35:51.106531 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:35:51.106513 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:36:00.143775 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:36:00.143741 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:36:11.628937 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:36:11.628900 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:36:20.732477 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:36:20.732396 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:36:31.441523 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:36:31.441489 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:36:41.234787 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:36:41.234751 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:36:49.136836 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:36:49.136797 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:36:58.430772 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:36:58.430735 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:37:21.431255 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:37:21.431225 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:37:30.737348 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:37:30.737311 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:37:39.944935 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:37:39.944900 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:37:53.131769 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:37:53.131683 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:38:02.233996 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:38:02.233962 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:38:10.141870 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:38:10.141830 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:38:18.434195 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:38:18.434161 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:38:28.330419 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:38:28.330382 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:38:37.332084 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:38:37.332034 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:38:46.945820 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:38:46.945783 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:38:56.433788 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:38:56.433758 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:39:09.541823 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:39:09.541785 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:39:18.749267 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:39:18.749187 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:39:28.637255 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:39:28.637210 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:39:41.237695 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:39:41.237657 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:39:53.040210 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:39:53.040169 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:40:02.252125 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:40:02.252089 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:40:15.839879 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:40:15.839837 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:40:25.636863 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:40:25.636828 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:40:35.041874 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:40:35.041833 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:40:44.240219 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:40:44.240136 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:40:47.246283 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:40:47.246249 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:40:51.128598 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:40:51.128567 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:40:51.131429 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:40:51.131404 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:40:53.132820 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:40:53.132770 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:40:56.816738 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:40:56.816703 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:05.118103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:05.118059 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:10.300864 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:10.300826 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:13.339870 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:13.339836 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:41:13.494661 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:13.494632 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:13.591696 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:13.591613 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:13.963264 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:13.963180 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:19.916298 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:19.916266 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:19.977727 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:19.977685 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:20.148816 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:20.148787 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:20.208557 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:20.208474 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:21.336696 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:21.336662 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:21.439687 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:21.439627 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:21.536909 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:21.536875 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:21.608321 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:21.608237 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:21.984484 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:21.984395 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:22.085309 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:22.085274 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:24.331873 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:24.331842 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:26.638167 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:26.638132 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:41:36.920204 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:36.920161 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:40.828762 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:40.828720 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:41:41.491616 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:41.491581 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:47.544876 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:47.544840 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:47.644743 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:47.644713 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:51.713312 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:51.713269 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:52.077261 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:52.077216 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:52.183114 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:52.183070 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:52.392004 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:52.391968 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:53.651051 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:53.651004 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:53.749465 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:53.749408 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:53.897515 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:53.897485 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:54.382699 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:54.382639 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:54.488495 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:54.488463 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:58.880232 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:58.880200 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:58.938336 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:58.938305 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:41:59.140009 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:41:59.139924 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:42:02.557521 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:02.557479 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:08.601426 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:08.601375 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:13.204999 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:13.204922 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:13.538160 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:13.538038 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:42:19.209823 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:19.209785 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:19.274605 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:19.274571 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:23.404878 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:23.404803 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:24.159905 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:24.159875 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:24.829588 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:24.829557 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:24.924769 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:24.924737 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:25.015286 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:25.015252 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:25.193742 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:25.193663 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:25.510922 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:25.510811 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:25.611241 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:25.611195 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:27.634631 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:27.634595 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:42:29.876344 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:29.876311 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:29.935755 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:29.935717 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:33.655605 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:33.655568 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:40.429856 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:40.429820 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:45.028715 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:45.028683 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:42:45.093553 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:45.093523 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:50.918192 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:50.918157 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:50.978879 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:50.978849 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:56.223102 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:56.223065 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:56.706181 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:56.706147 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:57.829176 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:57.829138 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:57.933284 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:57.933254 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:58.095632 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:58.095548 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:58.290660 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:58.290632 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:42:58.763559 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:42:58.763527 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:43:01.073832 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:43:01.073801 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:43:02.683008 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:43:02.682975 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:43:03.073299 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:43:03.073269 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:43:03.190581 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:43:03.190551 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:43:03.555166 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:43:03.555131 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:43:07.350150 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:43:07.350115 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:43:51.473654 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:43:51.473572 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:43:51.536851 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:43:51.536818 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:02.761700 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:02.761634 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:05.633168 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:05.633132 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:44:07.279380 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:07.279348 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:13.278085 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:13.278029 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:13.342224 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:13.342190 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:17.498211 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:17.498177 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:18.141183 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:18.141146 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:18.238594 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:18.238559 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:18.339387 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:18.339348 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:19.941353 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:19.941321 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:21.851638 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:21.851601 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:21.952129 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:21.952099 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:23.865580 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:23.865541 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:26.584873 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:26.584833 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:26.684870 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:26.684841 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:26.791302 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:26.791268 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:26.894809 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:26.894727 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:27.355160 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:27.355124 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:53.430598 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:53.430559 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:59.451152 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:59.451104 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:59.508387 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:59.508352 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:59.613283 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:59.613242 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:44:59.718263 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:44:59.718181 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:11.450590 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:11.450514 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:11.515500 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:11.515467 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:15.726407 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:15.726309 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:21.828257 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:21.828216 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:21.932984 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:21.932941 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:22.090533 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:22.090459 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:32.614974 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:32.614933 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:33.532468 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:33.532438 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:39.682703 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:39.682670 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:39.783529 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:39.783488 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:39.883830 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:39.883760 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:39.985296 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:39.985217 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:40.086997 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:40.086956 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:42.343924 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:42.343887 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:45:42.568668 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:42.568624 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:42.641678 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:42.641603 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:43.606806 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:43.606772 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:49.229721 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:49.229684 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:49.336236 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:49.336202 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:49.733724 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:49.733691 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:45:49.976913 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:49.976883 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:51.151455 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:51.151425 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:45:51.155974 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:51.155951 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:45:56.391328 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:56.391289 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:45:57.550780 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:45:57.550742 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:46:00.963324 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:00.963275 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:01.089890 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:01.089826 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:01.244354 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:01.244268 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:01.353571 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:01.353534 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:04.799562 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:04.799525 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:05.509373 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:05.509312 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:05.747233 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:05.747201 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:46:07.193327 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:07.193219 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:07.284729 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:07.284689 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:12.001576 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:12.001544 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:12.071033 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:12.071002 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:22.862240 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:22.862207 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:25.845263 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:25.845226 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:46:28.827329 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:28.827289 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:34.183024 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:34.182986 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:34.273325 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:34.273286 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:38.284500 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:38.284465 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:38.555017 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:38.554905 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:38.691492 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:38.691457 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:38.784989 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:38.784955 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:38.930727 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:38.930696 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:46.346680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:46.346605 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:46.448698 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:46.448668 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:51.686684 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:51.686644 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:51.787521 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:51.787470 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:58.768650 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:58.768603 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:59.259400 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:59.259360 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:59.378029 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:46:59.377989 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:02.566526 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:02.566492 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:02.666442 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:02.666413 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:03.157252 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:03.157221 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:03.270206 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:03.270173 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:03.465496 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:03.465411 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:03.565550 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:03.565518 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:06.705760 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:06.705722 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:47:07.174458 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:07.174425 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:21.705475 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.705442 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r"] Jul 14 19:47:21.705879 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.705718 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acce073-69a8-4e39-8528-5634d3459b1a" containerName="authorino" Jul 14 19:47:21.705879 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.705730 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acce073-69a8-4e39-8528-5634d3459b1a" containerName="authorino" Jul 14 19:47:21.705879 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.705791 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3acce073-69a8-4e39-8528-5634d3459b1a" containerName="authorino" Jul 14 19:47:21.708711 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.708694 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.711461 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.711438 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ea441188\"/\"openshift-service-ca.crt\"" Jul 14 19:47:21.711461 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.711458 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ea441188\"/\"kube-root-ca.crt\"" Jul 14 19:47:21.712550 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.712532 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ea441188\"/\"e2e-lifecycle-model-ea441188-kserve-self-signed-certs\"" Jul 14 19:47:21.712648 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.712532 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ea441188\"/\"default-dockercfg-nbrj6\"" Jul 14 19:47:21.719219 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.719198 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r"] Jul 14 19:47:21.745444 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.745415 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:21.845266 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.845230 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-dshm\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.845443 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.845351 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-home\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.845443 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.845383 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b58k\" (UniqueName: \"kubernetes.io/projected/56e3d2e2-b084-4ed1-a6b6-7817565c909a-kube-api-access-6b58k\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.845443 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.845403 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e3d2e2-b084-4ed1-a6b6-7817565c909a-tls-certs\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.845443 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.845428 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-model-cache\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.845571 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.845487 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-tmp-dir\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.946349 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.946313 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b58k\" (UniqueName: \"kubernetes.io/projected/56e3d2e2-b084-4ed1-a6b6-7817565c909a-kube-api-access-6b58k\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.946349 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.946351 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e3d2e2-b084-4ed1-a6b6-7817565c909a-tls-certs\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.946524 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.946377 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-model-cache\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.946524 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.946495 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-tmp-dir\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.946618 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.946557 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-dshm\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.946669 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.946647 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-home\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.946725 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.946696 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-model-cache\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.946888 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.946868 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-tmp-dir\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.946959 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.946898 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-home\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.948714 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.948694 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-dshm\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.948942 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.948925 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e3d2e2-b084-4ed1-a6b6-7817565c909a-tls-certs\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:21.963462 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:21.963398 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b58k\" (UniqueName: \"kubernetes.io/projected/56e3d2e2-b084-4ed1-a6b6-7817565c909a-kube-api-access-6b58k\") pod \"e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:22.018554 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:22.018510 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:22.152114 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:22.151980 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r"] Jul 14 19:47:22.154769 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:47:22.154746 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56e3d2e2_b084_4ed1_a6b6_7817565c909a.slice/crio-a5a3e7096a13c3d730663eabb3d54c0c77e2f01bd33d1d7d62e10aa50bd95f78 WatchSource:0}: Error finding container a5a3e7096a13c3d730663eabb3d54c0c77e2f01bd33d1d7d62e10aa50bd95f78: Status 404 returned error can't find the container with id a5a3e7096a13c3d730663eabb3d54c0c77e2f01bd33d1d7d62e10aa50bd95f78 Jul 14 19:47:22.586958 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:22.586921 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" event={"ID":"56e3d2e2-b084-4ed1-a6b6-7817565c909a","Type":"ContainerStarted","Data":"e4ac871c1835a01b791a64b7cb388157ba7818a355c11a0f04c980047d961ccb"} Jul 14 19:47:22.587159 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:22.586970 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" event={"ID":"56e3d2e2-b084-4ed1-a6b6-7817565c909a","Type":"ContainerStarted","Data":"a5a3e7096a13c3d730663eabb3d54c0c77e2f01bd33d1d7d62e10aa50bd95f78"} Jul 14 19:47:22.606713 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:22.606653 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" podStartSLOduration=1.6066376340000001 podStartE2EDuration="1.606637634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:22.604960317 +0000 UTC m=+2191.970030678" watchObservedRunningTime="2026-07-14 19:47:22.606637634 +0000 UTC m=+2191.971707994" Jul 14 19:47:23.742104 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:23.742067 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:47:32.018861 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:32.018805 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:32.018861 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:32.018874 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:32.031547 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:32.031522 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:32.639561 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:32.639534 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:47:37.641343 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:37.641303 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:47:51.511847 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:51.511805 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:47:55.026844 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:55.026804 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:47:56.069643 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:47:56.069610 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:01.054311 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:01.054272 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r"] Jul 14 19:48:01.107762 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:01.107724 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:16.074680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.074582 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" podUID="56e3d2e2-b084-4ed1-a6b6-7817565c909a" containerName="main" containerID="cri-o://e4ac871c1835a01b791a64b7cb388157ba7818a355c11a0f04c980047d961ccb" gracePeriod=45 Jul 14 19:48:16.309950 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.309926 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:48:16.392365 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.392336 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6b58k\" (UniqueName: \"kubernetes.io/projected/56e3d2e2-b084-4ed1-a6b6-7817565c909a-kube-api-access-6b58k\") pod \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " Jul 14 19:48:16.392550 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.392374 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-dshm\") pod \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " Jul 14 19:48:16.392550 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.392396 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e3d2e2-b084-4ed1-a6b6-7817565c909a-tls-certs\") pod \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " Jul 14 19:48:16.392550 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.392419 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-tmp-dir\") pod \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " Jul 14 19:48:16.392726 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.392555 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-model-cache\") pod \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " Jul 14 19:48:16.392726 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.392642 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-home\") pod \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\" (UID: \"56e3d2e2-b084-4ed1-a6b6-7817565c909a\") " Jul 14 19:48:16.392726 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.392652 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56e3d2e2-b084-4ed1-a6b6-7817565c909a" (UID: "56e3d2e2-b084-4ed1-a6b6-7817565c909a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:48:16.392884 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.392775 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-model-cache" (OuterVolumeSpecName: "model-cache") pod "56e3d2e2-b084-4ed1-a6b6-7817565c909a" (UID: "56e3d2e2-b084-4ed1-a6b6-7817565c909a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:48:16.392940 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.392899 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-model-cache\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:48:16.392940 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.392903 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-home" (OuterVolumeSpecName: "home") pod "56e3d2e2-b084-4ed1-a6b6-7817565c909a" (UID: "56e3d2e2-b084-4ed1-a6b6-7817565c909a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:48:16.392940 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.392920 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-tmp-dir\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:48:16.394603 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.394576 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-dshm" (OuterVolumeSpecName: "dshm") pod "56e3d2e2-b084-4ed1-a6b6-7817565c909a" (UID: "56e3d2e2-b084-4ed1-a6b6-7817565c909a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:48:16.394708 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.394601 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e3d2e2-b084-4ed1-a6b6-7817565c909a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56e3d2e2-b084-4ed1-a6b6-7817565c909a" (UID: "56e3d2e2-b084-4ed1-a6b6-7817565c909a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:48:16.394708 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.394613 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e3d2e2-b084-4ed1-a6b6-7817565c909a-kube-api-access-6b58k" (OuterVolumeSpecName: "kube-api-access-6b58k") pod "56e3d2e2-b084-4ed1-a6b6-7817565c909a" (UID: "56e3d2e2-b084-4ed1-a6b6-7817565c909a"). InnerVolumeSpecName "kube-api-access-6b58k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:48:16.493521 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.493497 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-home\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:48:16.493521 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.493520 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b58k\" (UniqueName: \"kubernetes.io/projected/56e3d2e2-b084-4ed1-a6b6-7817565c909a-kube-api-access-6b58k\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:48:16.493660 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.493535 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e3d2e2-b084-4ed1-a6b6-7817565c909a-dshm\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:48:16.493660 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.493547 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e3d2e2-b084-4ed1-a6b6-7817565c909a-tls-certs\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:48:16.812712 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.812614 2572 generic.go:356] "Generic (PLEG): container finished" podID="56e3d2e2-b084-4ed1-a6b6-7817565c909a" containerID="e4ac871c1835a01b791a64b7cb388157ba7818a355c11a0f04c980047d961ccb" exitCode=0 Jul 14 19:48:16.812712 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.812651 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" Jul 14 19:48:16.812899 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.812703 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" event={"ID":"56e3d2e2-b084-4ed1-a6b6-7817565c909a","Type":"ContainerDied","Data":"e4ac871c1835a01b791a64b7cb388157ba7818a355c11a0f04c980047d961ccb"} Jul 14 19:48:16.812899 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.812743 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r" event={"ID":"56e3d2e2-b084-4ed1-a6b6-7817565c909a","Type":"ContainerDied","Data":"a5a3e7096a13c3d730663eabb3d54c0c77e2f01bd33d1d7d62e10aa50bd95f78"} Jul 14 19:48:16.812899 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.812759 2572 scope.go:117] "RemoveContainer" containerID="e4ac871c1835a01b791a64b7cb388157ba7818a355c11a0f04c980047d961ccb" Jul 14 19:48:16.821478 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.821462 2572 scope.go:117] "RemoveContainer" containerID="e4ac871c1835a01b791a64b7cb388157ba7818a355c11a0f04c980047d961ccb" Jul 14 19:48:16.821746 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:48:16.821731 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ac871c1835a01b791a64b7cb388157ba7818a355c11a0f04c980047d961ccb\": container with ID starting with e4ac871c1835a01b791a64b7cb388157ba7818a355c11a0f04c980047d961ccb not found: ID does not exist" containerID="e4ac871c1835a01b791a64b7cb388157ba7818a355c11a0f04c980047d961ccb" Jul 14 19:48:16.821788 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.821754 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ac871c1835a01b791a64b7cb388157ba7818a355c11a0f04c980047d961ccb"} err="failed to get container status \"e4ac871c1835a01b791a64b7cb388157ba7818a355c11a0f04c980047d961ccb\": rpc error: code = NotFound desc = could not find container \"e4ac871c1835a01b791a64b7cb388157ba7818a355c11a0f04c980047d961ccb\": container with ID starting with e4ac871c1835a01b791a64b7cb388157ba7818a355c11a0f04c980047d961ccb not found: ID does not exist" Jul 14 19:48:16.848694 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.848669 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r"] Jul 14 19:48:16.861176 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:16.861154 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ea441188/e2e-lifecycle-model-ea441188-kserve-5bf6789dc5-vrr8r"] Jul 14 19:48:17.192656 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:17.192622 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56e3d2e2-b084-4ed1-a6b6-7817565c909a" path="/var/lib/kubelet/pods/56e3d2e2-b084-4ed1-a6b6-7817565c909a/volumes" Jul 14 19:48:37.347516 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:37.347483 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:38.432537 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:38.432498 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:44.646319 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:44.646280 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:59.630683 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:59.630647 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:48:59.713769 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:48:59.713742 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:08.584529 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:08.584493 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:08.838272 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:08.838181 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:10.029648 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:10.029610 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:10.127426 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:10.127395 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:10.224446 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:10.224411 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:11.201219 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:11.201187 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:11.281343 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:11.281311 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:11.346189 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:11.346161 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:12.550910 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:12.550869 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:49:14.171192 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:14.171156 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:19.187385 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:19.187345 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:19.288516 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:19.288468 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:19.893152 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:19.893117 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:21.260790 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:21.260752 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:23.872744 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:23.872710 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:23.974554 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:23.974525 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:24.534853 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:24.534820 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:49:26.190282 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:26.190252 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:26.293239 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:26.293205 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:26.391751 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:26.391722 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:36.960605 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:36.960567 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:37.326629 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:37.326593 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:49:58.355455 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:49:58.355347 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:04.744302 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:04.744262 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:04.844670 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:04.844641 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:05.074243 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:05.074157 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:05.500572 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:05.500538 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:05.727423 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:05.727386 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:08.445574 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:08.445538 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:08.530374 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:08.530339 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:09.106247 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:09.106203 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:09.210280 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:09.210238 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:09.304014 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:09.303983 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:09.404192 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:09.404110 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:12.976326 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:12.976292 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:13.367236 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:13.367207 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:13.687792 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:13.687710 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:14.096320 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:14.096288 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:16.097643 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:16.097606 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:50:16.430365 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:16.430276 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:16.534808 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:16.534774 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:16.711841 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:16.711757 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:16.893802 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:16.893771 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:17.225705 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:17.225665 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:17.320826 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:17.320795 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:20.979115 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:20.979075 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:26.775355 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:26.775324 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:26.881032 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:26.880996 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:30.342647 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:30.342607 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:50:31.147013 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:31.146979 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:31.248284 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:31.248252 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:35.410788 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.410752 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz"] Jul 14 19:50:35.411167 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.411037 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e3d2e2-b084-4ed1-a6b6-7817565c909a" containerName="main" Jul 14 19:50:35.411167 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.411063 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e3d2e2-b084-4ed1-a6b6-7817565c909a" containerName="main" Jul 14 19:50:35.411167 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.411132 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="56e3d2e2-b084-4ed1-a6b6-7817565c909a" containerName="main" Jul 14 19:50:35.413962 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.413947 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.417526 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.417494 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0c147a\"/\"kube-root-ca.crt\"" Jul 14 19:50:35.417526 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.417514 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0c147a\"/\"openshift-service-ca.crt\"" Jul 14 19:50:35.417690 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.417517 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0c147a\"/\"test-model-0c147a-kserve-self-signed-certs\"" Jul 14 19:50:35.417690 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.417556 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0c147a\"/\"default-dockercfg-zrzxg\"" Jul 14 19:50:35.426102 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.426083 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz"] Jul 14 19:50:35.436704 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.436680 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:35.448417 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.448395 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94bf0fee-b13e-4543-aef1-e9f21d1a6914-tls-certs\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.448520 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.448429 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-model-cache\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.448520 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.448458 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8j4j\" (UniqueName: \"kubernetes.io/projected/94bf0fee-b13e-4543-aef1-e9f21d1a6914-kube-api-access-h8j4j\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.448618 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.448534 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-tmp-dir\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.448618 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.448560 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-home\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.448618 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.448578 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-dshm\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.548924 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.548887 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-tmp-dir\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.548924 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.548927 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-home\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.549215 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.548944 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-dshm\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.549215 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.548962 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94bf0fee-b13e-4543-aef1-e9f21d1a6914-tls-certs\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.549215 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.548991 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-model-cache\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.549215 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.549036 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8j4j\" (UniqueName: \"kubernetes.io/projected/94bf0fee-b13e-4543-aef1-e9f21d1a6914-kube-api-access-h8j4j\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.549422 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.549391 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-tmp-dir\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.549422 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.549414 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-home\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.549506 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.549448 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-model-cache\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.551377 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.551341 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-dshm\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.551491 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.551448 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94bf0fee-b13e-4543-aef1-e9f21d1a6914-tls-certs\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.558159 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.558138 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8j4j\" (UniqueName: \"kubernetes.io/projected/94bf0fee-b13e-4543-aef1-e9f21d1a6914-kube-api-access-h8j4j\") pod \"test-model-0c147a-kserve-65874d48d9-2x4nz\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.723223 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.723120 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:35.857344 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:35.857283 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz"] Jul 14 19:50:35.859640 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:50:35.859607 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94bf0fee_b13e_4543_aef1_e9f21d1a6914.slice/crio-90cdc970c0db288ca2f3e29cdac3382510e392949d0aa7e12742243124658a7a WatchSource:0}: Error finding container 90cdc970c0db288ca2f3e29cdac3382510e392949d0aa7e12742243124658a7a: Status 404 returned error can't find the container with id 90cdc970c0db288ca2f3e29cdac3382510e392949d0aa7e12742243124658a7a Jul 14 19:50:36.376966 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:36.376929 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" event={"ID":"94bf0fee-b13e-4543-aef1-e9f21d1a6914","Type":"ContainerStarted","Data":"c12d9e57d06c5597e16295440599dc707d98fe153ec1c63b4ee9d64ba032fe43"} Jul 14 19:50:36.376966 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:36.376977 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" event={"ID":"94bf0fee-b13e-4543-aef1-e9f21d1a6914","Type":"ContainerStarted","Data":"90cdc970c0db288ca2f3e29cdac3382510e392949d0aa7e12742243124658a7a"} Jul 14 19:50:36.395491 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:36.395436 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" podStartSLOduration=1.3954213100000001 podStartE2EDuration="1.39542131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:50:36.394419205 +0000 UTC m=+2385.759489570" watchObservedRunningTime="2026-07-14 19:50:36.39542131 +0000 UTC m=+2385.760491673" Jul 14 19:50:39.088583 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:39.088550 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:39.187974 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:39.187944 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:40.641095 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:40.641058 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:50:45.724244 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:45.724193 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:45.724733 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:45.724263 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:45.736668 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:45.736639 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:46.430241 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:46.430214 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:50:50.209054 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.209018 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv"] Jul 14 19:50:50.212326 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.212309 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.215941 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.215919 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7fc5dd\"/\"kube-root-ca.crt\"" Jul 14 19:50:50.216074 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.215964 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7fc5dd\"/\"default-dockercfg-2rrcl\"" Jul 14 19:50:50.216074 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.215966 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7fc5dd\"/\"openshift-service-ca.crt\"" Jul 14 19:50:50.216074 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.216029 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7fc5dd\"/\"test-model-7fc5dd-kserve-self-signed-certs\"" Jul 14 19:50:50.222856 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.222834 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv"] Jul 14 19:50:50.247502 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.247470 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-model-cache\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.247667 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.247507 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-home\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.247667 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.247535 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qs72\" (UniqueName: \"kubernetes.io/projected/0e1d65a8-2840-4acd-b28a-2619bb0e5433-kube-api-access-8qs72\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.247667 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.247592 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-dshm\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.247667 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.247647 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1d65a8-2840-4acd-b28a-2619bb0e5433-tls-certs\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.247796 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.247723 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-tmp-dir\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.305608 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.305575 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:50:50.348457 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.348418 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qs72\" (UniqueName: \"kubernetes.io/projected/0e1d65a8-2840-4acd-b28a-2619bb0e5433-kube-api-access-8qs72\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.348457 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.348460 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-dshm\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.348711 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.348492 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1d65a8-2840-4acd-b28a-2619bb0e5433-tls-certs\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.348711 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.348556 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-tmp-dir\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.348711 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.348592 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-model-cache\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.348711 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.348628 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-home\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.348999 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.348974 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-tmp-dir\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.348999 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.348992 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-model-cache\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.349168 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.349010 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-home\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.350782 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.350755 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-dshm\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.351058 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.351028 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1d65a8-2840-4acd-b28a-2619bb0e5433-tls-certs\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.357521 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.357491 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qs72\" (UniqueName: \"kubernetes.io/projected/0e1d65a8-2840-4acd-b28a-2619bb0e5433-kube-api-access-8qs72\") pod \"test-model-7fc5dd-kserve-6cb649b9c9-d58cv\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.521412 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.521327 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:50:50.656287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:50.656241 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv"] Jul 14 19:50:50.662460 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:50:50.662433 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e1d65a8_2840_4acd_b28a_2619bb0e5433.slice/crio-d61a27c8132e381e4a3d6cd1c65b49025f249a33729b3c84ba0d357705ebe156 WatchSource:0}: Error finding container d61a27c8132e381e4a3d6cd1c65b49025f249a33729b3c84ba0d357705ebe156: Status 404 returned error can't find the container with id d61a27c8132e381e4a3d6cd1c65b49025f249a33729b3c84ba0d357705ebe156 Jul 14 19:50:51.175157 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:51.175130 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:50:51.180407 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:51.180389 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:50:51.440412 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:51.440327 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" event={"ID":"0e1d65a8-2840-4acd-b28a-2619bb0e5433","Type":"ContainerStarted","Data":"32b3e470a7e76e3c4c076b4307465c543ed6d990fd01e7197340521c5cda6132"} Jul 14 19:50:51.440412 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:51.440377 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" event={"ID":"0e1d65a8-2840-4acd-b28a-2619bb0e5433","Type":"ContainerStarted","Data":"d61a27c8132e381e4a3d6cd1c65b49025f249a33729b3c84ba0d357705ebe156"} Jul 14 19:50:51.459246 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:51.459192 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" podStartSLOduration=1.459176886 podStartE2EDuration="1.459176886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:50:51.458342811 +0000 UTC m=+2400.823413179" watchObservedRunningTime="2026-07-14 19:50:51.459176886 +0000 UTC m=+2400.824247225" Jul 14 19:50:57.839237 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:50:57.839197 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:51:00.079301 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:00.079256 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:00.522259 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:00.522175 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:51:00.522259 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:00.522219 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:51:00.534530 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:00.534506 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:51:01.498248 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:01.498219 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:51:14.595283 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:14.595201 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz"] Jul 14 19:51:14.622226 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:14.622190 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:15.505320 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:15.505285 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv"] Jul 14 19:51:15.941478 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:15.941443 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:16.280978 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:16.280897 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:16.416771 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:16.416737 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:19.946340 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:19.946308 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:51:26.829719 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:26.829661 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:29.615717 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:29.615654 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" podUID="94bf0fee-b13e-4543-aef1-e9f21d1a6914" containerName="main" containerID="cri-o://c12d9e57d06c5597e16295440599dc707d98fe153ec1c63b4ee9d64ba032fe43" gracePeriod=45 Jul 14 19:51:29.860397 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:29.860371 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:51:29.979287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:29.979190 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94bf0fee-b13e-4543-aef1-e9f21d1a6914-tls-certs\") pod \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " Jul 14 19:51:29.979287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:29.979259 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8j4j\" (UniqueName: \"kubernetes.io/projected/94bf0fee-b13e-4543-aef1-e9f21d1a6914-kube-api-access-h8j4j\") pod \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " Jul 14 19:51:29.979287 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:29.979280 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-model-cache\") pod \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " Jul 14 19:51:29.979594 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:29.979298 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-tmp-dir\") pod \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " Jul 14 19:51:29.979594 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:29.979332 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-dshm\") pod \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " Jul 14 19:51:29.979594 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:29.979372 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-home\") pod \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\" (UID: \"94bf0fee-b13e-4543-aef1-e9f21d1a6914\") " Jul 14 19:51:29.979735 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:29.979620 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-model-cache" (OuterVolumeSpecName: "model-cache") pod "94bf0fee-b13e-4543-aef1-e9f21d1a6914" (UID: "94bf0fee-b13e-4543-aef1-e9f21d1a6914"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:29.979735 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:29.979636 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94bf0fee-b13e-4543-aef1-e9f21d1a6914" (UID: "94bf0fee-b13e-4543-aef1-e9f21d1a6914"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:29.979735 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:29.979701 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-home" (OuterVolumeSpecName: "home") pod "94bf0fee-b13e-4543-aef1-e9f21d1a6914" (UID: "94bf0fee-b13e-4543-aef1-e9f21d1a6914"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:29.981582 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:29.981549 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94bf0fee-b13e-4543-aef1-e9f21d1a6914-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94bf0fee-b13e-4543-aef1-e9f21d1a6914" (UID: "94bf0fee-b13e-4543-aef1-e9f21d1a6914"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:51:29.981582 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:29.981573 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-dshm" (OuterVolumeSpecName: "dshm") pod "94bf0fee-b13e-4543-aef1-e9f21d1a6914" (UID: "94bf0fee-b13e-4543-aef1-e9f21d1a6914"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:29.981731 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:29.981644 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bf0fee-b13e-4543-aef1-e9f21d1a6914-kube-api-access-h8j4j" (OuterVolumeSpecName: "kube-api-access-h8j4j") pod "94bf0fee-b13e-4543-aef1-e9f21d1a6914" (UID: "94bf0fee-b13e-4543-aef1-e9f21d1a6914"). InnerVolumeSpecName "kube-api-access-h8j4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:51:30.080748 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.080715 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-home\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:51:30.080748 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.080741 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94bf0fee-b13e-4543-aef1-e9f21d1a6914-tls-certs\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:51:30.080748 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.080753 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8j4j\" (UniqueName: \"kubernetes.io/projected/94bf0fee-b13e-4543-aef1-e9f21d1a6914-kube-api-access-h8j4j\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:51:30.080983 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.080762 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-model-cache\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:51:30.080983 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.080770 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-tmp-dir\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:51:30.080983 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.080777 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94bf0fee-b13e-4543-aef1-e9f21d1a6914-dshm\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:51:30.524798 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.524744 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" podUID="0e1d65a8-2840-4acd-b28a-2619bb0e5433" containerName="main" containerID="cri-o://32b3e470a7e76e3c4c076b4307465c543ed6d990fd01e7197340521c5cda6132" gracePeriod=45 Jul 14 19:51:30.612628 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.612600 2572 generic.go:356] "Generic (PLEG): container finished" podID="94bf0fee-b13e-4543-aef1-e9f21d1a6914" containerID="c12d9e57d06c5597e16295440599dc707d98fe153ec1c63b4ee9d64ba032fe43" exitCode=0 Jul 14 19:51:30.612789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.612644 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" Jul 14 19:51:30.612789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.612681 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" event={"ID":"94bf0fee-b13e-4543-aef1-e9f21d1a6914","Type":"ContainerDied","Data":"c12d9e57d06c5597e16295440599dc707d98fe153ec1c63b4ee9d64ba032fe43"} Jul 14 19:51:30.612789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.612714 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz" event={"ID":"94bf0fee-b13e-4543-aef1-e9f21d1a6914","Type":"ContainerDied","Data":"90cdc970c0db288ca2f3e29cdac3382510e392949d0aa7e12742243124658a7a"} Jul 14 19:51:30.612789 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.612730 2572 scope.go:117] "RemoveContainer" containerID="c12d9e57d06c5597e16295440599dc707d98fe153ec1c63b4ee9d64ba032fe43" Jul 14 19:51:30.632625 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.632593 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz"] Jul 14 19:51:30.635785 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.635649 2572 scope.go:117] "RemoveContainer" containerID="c12d9e57d06c5597e16295440599dc707d98fe153ec1c63b4ee9d64ba032fe43" Jul 14 19:51:30.636003 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:51:30.635979 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12d9e57d06c5597e16295440599dc707d98fe153ec1c63b4ee9d64ba032fe43\": container with ID starting with c12d9e57d06c5597e16295440599dc707d98fe153ec1c63b4ee9d64ba032fe43 not found: ID does not exist" containerID="c12d9e57d06c5597e16295440599dc707d98fe153ec1c63b4ee9d64ba032fe43" Jul 14 19:51:30.636067 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.636014 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12d9e57d06c5597e16295440599dc707d98fe153ec1c63b4ee9d64ba032fe43"} err="failed to get container status \"c12d9e57d06c5597e16295440599dc707d98fe153ec1c63b4ee9d64ba032fe43\": rpc error: code = NotFound desc = could not find container \"c12d9e57d06c5597e16295440599dc707d98fe153ec1c63b4ee9d64ba032fe43\": container with ID starting with c12d9e57d06c5597e16295440599dc707d98fe153ec1c63b4ee9d64ba032fe43 not found: ID does not exist" Jul 14 19:51:30.639409 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.639383 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0c147a/test-model-0c147a-kserve-65874d48d9-2x4nz"] Jul 14 19:51:30.731680 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.731652 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:30.771572 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.771551 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:51:30.887272 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.887241 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1d65a8-2840-4acd-b28a-2619bb0e5433-tls-certs\") pod \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " Jul 14 19:51:30.887461 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.887284 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-home\") pod \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " Jul 14 19:51:30.887461 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.887304 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-model-cache\") pod \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " Jul 14 19:51:30.887461 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.887395 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qs72\" (UniqueName: \"kubernetes.io/projected/0e1d65a8-2840-4acd-b28a-2619bb0e5433-kube-api-access-8qs72\") pod \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " Jul 14 19:51:30.887461 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.887424 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-tmp-dir\") pod \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " Jul 14 19:51:30.887461 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.887443 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-dshm\") pod \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\" (UID: \"0e1d65a8-2840-4acd-b28a-2619bb0e5433\") " Jul 14 19:51:30.887693 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.887575 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-home" (OuterVolumeSpecName: "home") pod "0e1d65a8-2840-4acd-b28a-2619bb0e5433" (UID: "0e1d65a8-2840-4acd-b28a-2619bb0e5433"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:30.887693 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.887632 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e1d65a8-2840-4acd-b28a-2619bb0e5433" (UID: "0e1d65a8-2840-4acd-b28a-2619bb0e5433"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:30.887805 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.887720 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e1d65a8-2840-4acd-b28a-2619bb0e5433" (UID: "0e1d65a8-2840-4acd-b28a-2619bb0e5433"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:30.889586 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.889560 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-dshm" (OuterVolumeSpecName: "dshm") pod "0e1d65a8-2840-4acd-b28a-2619bb0e5433" (UID: "0e1d65a8-2840-4acd-b28a-2619bb0e5433"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:30.889586 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.889570 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e1d65a8-2840-4acd-b28a-2619bb0e5433-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e1d65a8-2840-4acd-b28a-2619bb0e5433" (UID: "0e1d65a8-2840-4acd-b28a-2619bb0e5433"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:51:30.889735 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.889605 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1d65a8-2840-4acd-b28a-2619bb0e5433-kube-api-access-8qs72" (OuterVolumeSpecName: "kube-api-access-8qs72") pod "0e1d65a8-2840-4acd-b28a-2619bb0e5433" (UID: "0e1d65a8-2840-4acd-b28a-2619bb0e5433"). InnerVolumeSpecName "kube-api-access-8qs72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:51:30.988865 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.988824 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-dshm\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:51:30.988865 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.988856 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1d65a8-2840-4acd-b28a-2619bb0e5433-tls-certs\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:51:30.988865 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.988866 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-home\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:51:30.988865 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.988873 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-model-cache\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:51:30.989158 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.988882 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qs72\" (UniqueName: \"kubernetes.io/projected/0e1d65a8-2840-4acd-b28a-2619bb0e5433-kube-api-access-8qs72\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:51:30.989158 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:30.988891 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1d65a8-2840-4acd-b28a-2619bb0e5433-tmp-dir\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:51:31.188865 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:31.188783 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94bf0fee-b13e-4543-aef1-e9f21d1a6914" path="/var/lib/kubelet/pods/94bf0fee-b13e-4543-aef1-e9f21d1a6914/volumes" Jul 14 19:51:31.618166 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:31.618132 2572 generic.go:356] "Generic (PLEG): container finished" podID="0e1d65a8-2840-4acd-b28a-2619bb0e5433" containerID="32b3e470a7e76e3c4c076b4307465c543ed6d990fd01e7197340521c5cda6132" exitCode=0 Jul 14 19:51:31.618351 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:31.618193 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" Jul 14 19:51:31.618351 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:31.618222 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" event={"ID":"0e1d65a8-2840-4acd-b28a-2619bb0e5433","Type":"ContainerDied","Data":"32b3e470a7e76e3c4c076b4307465c543ed6d990fd01e7197340521c5cda6132"} Jul 14 19:51:31.618351 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:31.618272 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv" event={"ID":"0e1d65a8-2840-4acd-b28a-2619bb0e5433","Type":"ContainerDied","Data":"d61a27c8132e381e4a3d6cd1c65b49025f249a33729b3c84ba0d357705ebe156"} Jul 14 19:51:31.618351 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:31.618297 2572 scope.go:117] "RemoveContainer" containerID="32b3e470a7e76e3c4c076b4307465c543ed6d990fd01e7197340521c5cda6132" Jul 14 19:51:31.626417 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:31.626399 2572 scope.go:117] "RemoveContainer" containerID="32b3e470a7e76e3c4c076b4307465c543ed6d990fd01e7197340521c5cda6132" Jul 14 19:51:31.626683 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:51:31.626664 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b3e470a7e76e3c4c076b4307465c543ed6d990fd01e7197340521c5cda6132\": container with ID starting with 32b3e470a7e76e3c4c076b4307465c543ed6d990fd01e7197340521c5cda6132 not found: ID does not exist" containerID="32b3e470a7e76e3c4c076b4307465c543ed6d990fd01e7197340521c5cda6132" Jul 14 19:51:31.626747 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:31.626697 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b3e470a7e76e3c4c076b4307465c543ed6d990fd01e7197340521c5cda6132"} err="failed to get container status \"32b3e470a7e76e3c4c076b4307465c543ed6d990fd01e7197340521c5cda6132\": rpc error: code = NotFound desc = could not find container \"32b3e470a7e76e3c4c076b4307465c543ed6d990fd01e7197340521c5cda6132\": container with ID starting with 32b3e470a7e76e3c4c076b4307465c543ed6d990fd01e7197340521c5cda6132 not found: ID does not exist" Jul 14 19:51:31.634773 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:31.634747 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv"] Jul 14 19:51:31.638599 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:31.638579 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7fc5dd/test-model-7fc5dd-kserve-6cb649b9c9-d58cv"] Jul 14 19:51:33.187973 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:33.187939 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e1d65a8-2840-4acd-b28a-2619bb0e5433" path="/var/lib/kubelet/pods/0e1d65a8-2840-4acd-b28a-2619bb0e5433/volumes" Jul 14 19:51:37.108267 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:37.108232 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:37.211061 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:37.211000 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:37.245452 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:37.245416 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:51:41.469422 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:41.469362 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:42.060675 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:42.060642 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:42.750774 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:42.750740 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:42.851342 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:42.851308 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:52.292848 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:52.292815 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:51:55.441658 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:55.441623 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:51:56.926096 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:51:56.926055 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:03.515799 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:03.515764 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:03.627010 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:03.626981 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:07.640766 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:07.640724 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:07.878147 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:07.878114 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:08.278214 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:08.278169 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:08.607285 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:08.607250 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:09.325459 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:09.325412 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:11.175130 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:11.175097 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:11.267662 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:11.267621 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:11.415447 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:11.415412 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:11.685435 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:11.685394 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:12.057994 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:12.057904 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:12.158623 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:12.158594 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:15.821766 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:15.821728 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:16.206836 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:16.206809 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:16.544319 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:16.544234 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:16.995486 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:16.995453 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:18.538253 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:18.538221 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:52:19.310713 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:19.310679 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:19.395908 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:19.395879 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:19.964881 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:19.964837 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:20.059104 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:20.059065 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:20.164255 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:20.164217 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:23.713614 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:23.713582 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:32.503276 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:32.503243 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:32.600336 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:32.600306 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:34.639646 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:34.639610 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:52:40.812393 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:40.812356 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:41.362874 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:41.362837 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:42.582110 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:42.581988 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:42.681509 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:42.681478 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:42.843616 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:42.843541 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:44.776862 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:44.776819 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:44.879247 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:44.879215 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:45.536028 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:45.535992 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:52:47.002641 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:47.002583 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:53.043754 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:53.043713 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:53.143699 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:53.143654 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:57.746900 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:57.746865 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:58.015005 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:58.014902 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:58.815285 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:58.815250 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:52:58.911193 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:52:58.911164 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:08.728320 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:08.728283 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:11.953751 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:11.953717 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:53:13.247838 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:13.247801 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:19.524761 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:19.524732 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:19.623465 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:19.623433 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:24.235820 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:24.235769 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:24.335699 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:24.335653 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:24.618283 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:24.618245 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:24.959084 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:24.958977 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:25.435776 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:25.435740 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:27.591640 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:27.591609 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:27.675843 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:27.675804 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:27.802666 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:27.802632 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:28.261024 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:28.260992 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:28.362030 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:28.361999 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:32.097353 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:32.097316 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:32.477303 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:32.477217 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:32.843798 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:32.843765 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:33.069927 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:33.069890 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:33.436992 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:33.436953 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:53:36.110386 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:36.110350 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:36.209329 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:36.209296 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:36.477154 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:36.477073 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:36.577161 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:36.577118 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:40.157274 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:40.157237 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:41.271378 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:41.271336 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:41.652024 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:41.651991 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:49.188503 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:49.188475 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:49.281650 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:49.281606 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:50.149690 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:50.149655 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:53:58.276496 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.276460 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:58.375868 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.375829 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:58.712970 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.712939 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb"] Jul 14 19:53:58.713327 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.713309 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94bf0fee-b13e-4543-aef1-e9f21d1a6914" containerName="main" Jul 14 19:53:58.713416 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.713330 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="94bf0fee-b13e-4543-aef1-e9f21d1a6914" containerName="main" Jul 14 19:53:58.713416 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.713343 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e1d65a8-2840-4acd-b28a-2619bb0e5433" containerName="main" Jul 14 19:53:58.713416 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.713352 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e1d65a8-2840-4acd-b28a-2619bb0e5433" containerName="main" Jul 14 19:53:58.713564 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.713436 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e1d65a8-2840-4acd-b28a-2619bb0e5433" containerName="main" Jul 14 19:53:58.713564 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.713452 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="94bf0fee-b13e-4543-aef1-e9f21d1a6914" containerName="main" Jul 14 19:53:58.716366 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.716336 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.718795 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.718770 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-70659c\"/\"kube-root-ca.crt\"" Jul 14 19:53:58.719792 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.719770 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-70659c\"/\"default-dockercfg-s5ftl\"" Jul 14 19:53:58.719903 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.719797 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-70659c\"/\"openshift-service-ca.crt\"" Jul 14 19:53:58.719903 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.719854 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-70659c\"/\"auth-test-model-70659c-kserve-self-signed-certs\"" Jul 14 19:53:58.724927 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.724897 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb"] Jul 14 19:53:58.745574 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.745549 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:53:58.801712 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.801670 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-tmp-dir\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.801896 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.801756 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfwqq\" (UniqueName: \"kubernetes.io/projected/48b9132d-b7c7-4788-ac38-05ea322a7683-kube-api-access-zfwqq\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.801896 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.801793 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48b9132d-b7c7-4788-ac38-05ea322a7683-tls-certs\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.801896 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.801850 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-model-cache\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.801896 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.801886 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-home\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.802086 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.801907 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-dshm\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.902515 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.902476 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfwqq\" (UniqueName: \"kubernetes.io/projected/48b9132d-b7c7-4788-ac38-05ea322a7683-kube-api-access-zfwqq\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.902515 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.902518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48b9132d-b7c7-4788-ac38-05ea322a7683-tls-certs\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.902735 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.902540 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-model-cache\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.902735 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.902562 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-home\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.902735 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.902613 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-dshm\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.902735 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.902677 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-tmp-dir\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.902943 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.902905 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-home\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.903002 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.902965 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-tmp-dir\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.903085 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.903037 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-model-cache\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.904903 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.904882 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-dshm\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.905084 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.905068 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48b9132d-b7c7-4788-ac38-05ea322a7683-tls-certs\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:58.910151 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:58.910128 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfwqq\" (UniqueName: \"kubernetes.io/projected/48b9132d-b7c7-4788-ac38-05ea322a7683-kube-api-access-zfwqq\") pod \"auth-test-model-70659c-kserve-679bf587bc-hrvwb\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:59.025954 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:59.025866 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:53:59.155394 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:59.155360 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb"] Jul 14 19:53:59.157782 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:53:59.157753 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48b9132d_b7c7_4788_ac38_05ea322a7683.slice/crio-397f58a2f959a6d22897c887a87d615809882c58c1dd97e7efc78cfcc2714b2d WatchSource:0}: Error finding container 397f58a2f959a6d22897c887a87d615809882c58c1dd97e7efc78cfcc2714b2d: Status 404 returned error can't find the container with id 397f58a2f959a6d22897c887a87d615809882c58c1dd97e7efc78cfcc2714b2d Jul 14 19:53:59.232841 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:59.232805 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" event={"ID":"48b9132d-b7c7-4788-ac38-05ea322a7683","Type":"ContainerStarted","Data":"6c0cb027331947c96ddec385e6e2ca805bf480f0e2c43a1db6b383c216ce5fba"} Jul 14 19:53:59.233028 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:53:59.232862 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" event={"ID":"48b9132d-b7c7-4788-ac38-05ea322a7683","Type":"ContainerStarted","Data":"397f58a2f959a6d22897c887a87d615809882c58c1dd97e7efc78cfcc2714b2d"} Jul 14 19:54:02.635671 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:02.635616 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" podStartSLOduration=4.635599778 podStartE2EDuration="4.635599778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:53:59.282432795 +0000 UTC m=+2588.647503156" watchObservedRunningTime="2026-07-14 19:54:02.635599778 +0000 UTC m=+2592.000670172" Jul 14 19:54:02.636865 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:02.636847 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:54:09.026663 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:09.026605 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:54:09.026663 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:09.026679 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:54:09.040463 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:09.040429 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:54:09.285779 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:09.285685 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:54:14.303831 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.303755 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w"] Jul 14 19:54:14.307114 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.307094 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.310958 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.310918 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-356b53\"/\"openshift-service-ca.crt\"" Jul 14 19:54:14.310958 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.310939 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-356b53\"/\"auth-test-model-356b53-kserve-self-signed-certs\"" Jul 14 19:54:14.311192 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.310939 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-356b53\"/\"default-dockercfg-d22z5\"" Jul 14 19:54:14.311192 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.311083 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-356b53\"/\"kube-root-ca.crt\"" Jul 14 19:54:14.316857 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.316833 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w"] Jul 14 19:54:14.435839 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.435781 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0716391-d5a6-4410-9b13-4577a018144d-tls-certs\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.436028 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.435866 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-dshm\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.436028 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.435897 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-home\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.436028 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.435925 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-model-cache\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.436028 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.435945 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-tmp-dir\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.436028 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.435968 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c4hl\" (UniqueName: \"kubernetes.io/projected/f0716391-d5a6-4410-9b13-4577a018144d-kube-api-access-4c4hl\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.536976 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.536929 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-dshm\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.536976 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.536979 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-home\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.537258 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.537001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-model-cache\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.537258 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.537018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-tmp-dir\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.537258 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.537160 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c4hl\" (UniqueName: \"kubernetes.io/projected/f0716391-d5a6-4410-9b13-4577a018144d-kube-api-access-4c4hl\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.537428 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.537275 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0716391-d5a6-4410-9b13-4577a018144d-tls-certs\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.537428 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.537410 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-home\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.537503 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.537461 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-tmp-dir\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.537540 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.537500 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-model-cache\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.539325 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.539296 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-dshm\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.539568 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.539549 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0716391-d5a6-4410-9b13-4577a018144d-tls-certs\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.546868 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.546838 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c4hl\" (UniqueName: \"kubernetes.io/projected/f0716391-d5a6-4410-9b13-4577a018144d-kube-api-access-4c4hl\") pod \"auth-test-model-356b53-kserve-5685d5776c-lbc2w\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.618106 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.618064 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:14.751989 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:14.751877 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w"] Jul 14 19:54:14.754509 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:54:14.754483 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0716391_d5a6_4410_9b13_4577a018144d.slice/crio-d9c6e5dc7b62f2c0e08ad6b7a5fad432367a7975faf47744ea3623b91b4eb10c WatchSource:0}: Error finding container d9c6e5dc7b62f2c0e08ad6b7a5fad432367a7975faf47744ea3623b91b4eb10c: Status 404 returned error can't find the container with id d9c6e5dc7b62f2c0e08ad6b7a5fad432367a7975faf47744ea3623b91b4eb10c Jul 14 19:54:15.216557 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:15.216524 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:15.300836 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:15.300803 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" event={"ID":"f0716391-d5a6-4410-9b13-4577a018144d","Type":"ContainerStarted","Data":"8830072d9e72c21e4541dd197937a924fe20f4fbcffa5beb6b90bd28de177a79"} Jul 14 19:54:15.301014 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:15.300845 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" event={"ID":"f0716391-d5a6-4410-9b13-4577a018144d","Type":"ContainerStarted","Data":"d9c6e5dc7b62f2c0e08ad6b7a5fad432367a7975faf47744ea3623b91b4eb10c"} Jul 14 19:54:15.317872 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:15.317821 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" podStartSLOduration=1.317802286 podStartE2EDuration="1.317802286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:54:15.316609609 +0000 UTC m=+2604.681679972" watchObservedRunningTime="2026-07-14 19:54:15.317802286 +0000 UTC m=+2604.682872648" Jul 14 19:54:17.733419 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:17.733390 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:54:23.550951 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:23.550918 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:23.650723 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:23.650691 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:54:24.618795 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:24.618759 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:24.619213 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:24.619066 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:24.631173 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:24.631157 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:25.352607 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:25.352579 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:54:32.529402 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:32.529371 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:54:50.825606 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:54:50.825569 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:55:07.625627 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:55:07.625592 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:55:19.413251 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:55:19.413208 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:55:24.826471 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:55:24.826435 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:55:40.723602 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:55:40.723566 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:55:51.199540 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:55:51.199468 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:55:51.204586 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:55:51.204565 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 19:55:59.023624 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:55:59.023584 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:56:08.991873 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:08.991835 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:56:12.939358 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:12.939324 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb"] Jul 14 19:56:13.010615 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:13.010574 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:56:13.806299 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:13.806258 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w"] Jul 14 19:56:13.841755 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:13.841721 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:56:17.334058 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:17.334006 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:56:20.299013 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.298982 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts"] Jul 14 19:56:20.302418 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.302400 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.305102 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.305083 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-70659c\"/\"sub-test-model-70659c-kserve-self-signed-certs\"" Jul 14 19:56:20.312729 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.312705 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts"] Jul 14 19:56:20.324440 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.324415 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:56:20.383189 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.383151 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-model-cache\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.383371 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.383202 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d5t5\" (UniqueName: \"kubernetes.io/projected/afa9fa27-e2c1-41d6-a898-ec0b936280bf-kube-api-access-6d5t5\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.383371 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.383276 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa9fa27-e2c1-41d6-a898-ec0b936280bf-tls-certs\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.383371 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.383332 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-home\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.383371 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.383362 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-tmp-dir\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.383535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.383386 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-dshm\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.484788 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.484751 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-model-cache\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.484983 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.484796 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d5t5\" (UniqueName: \"kubernetes.io/projected/afa9fa27-e2c1-41d6-a898-ec0b936280bf-kube-api-access-6d5t5\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.484983 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.484825 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa9fa27-e2c1-41d6-a898-ec0b936280bf-tls-certs\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.484983 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.484860 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-home\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.484983 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.484879 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-tmp-dir\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.484983 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.484899 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-dshm\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.485323 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.485259 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-model-cache\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.485380 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.485334 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-tmp-dir\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.485380 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.485344 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-home\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.487249 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.487216 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-dshm\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.487503 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.487485 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa9fa27-e2c1-41d6-a898-ec0b936280bf-tls-certs\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.493415 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.493393 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d5t5\" (UniqueName: \"kubernetes.io/projected/afa9fa27-e2c1-41d6-a898-ec0b936280bf-kube-api-access-6d5t5\") pod \"sub-test-model-70659c-kserve-7967f95659-j4gts\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.611892 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.611850 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:20.744815 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.744788 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts"] Jul 14 19:56:20.747089 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:56:20.747056 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafa9fa27_e2c1_41d6_a898_ec0b936280bf.slice/crio-d862fa2f75cc82b6aa036ee85cc13e650e0284937849bbc246f8d28b7e3fade9 WatchSource:0}: Error finding container d862fa2f75cc82b6aa036ee85cc13e650e0284937849bbc246f8d28b7e3fade9: Status 404 returned error can't find the container with id d862fa2f75cc82b6aa036ee85cc13e650e0284937849bbc246f8d28b7e3fade9 Jul 14 19:56:20.838739 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.838705 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" event={"ID":"afa9fa27-e2c1-41d6-a898-ec0b936280bf","Type":"ContainerStarted","Data":"056d1a6c9c19f15e26f54bd6f0b488817af39d2795fa83cd35c7fd3b819dbae6"} Jul 14 19:56:20.838921 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.838750 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" event={"ID":"afa9fa27-e2c1-41d6-a898-ec0b936280bf","Type":"ContainerStarted","Data":"d862fa2f75cc82b6aa036ee85cc13e650e0284937849bbc246f8d28b7e3fade9"} Jul 14 19:56:20.857935 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:20.857888 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" podStartSLOduration=0.8578723 podStartE2EDuration="857.8723ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:56:20.856819037 +0000 UTC m=+2730.221889399" watchObservedRunningTime="2026-07-14 19:56:20.8578723 +0000 UTC m=+2730.222942660" Jul 14 19:56:27.958860 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:27.958813 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" podUID="48b9132d-b7c7-4788-ac38-05ea322a7683" containerName="main" containerID="cri-o://6c0cb027331947c96ddec385e6e2ca805bf480f0e2c43a1db6b383c216ce5fba" gracePeriod=45 Jul 14 19:56:28.220842 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.220783 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:56:28.352312 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.352280 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-home\") pod \"48b9132d-b7c7-4788-ac38-05ea322a7683\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " Jul 14 19:56:28.352506 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.352340 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfwqq\" (UniqueName: \"kubernetes.io/projected/48b9132d-b7c7-4788-ac38-05ea322a7683-kube-api-access-zfwqq\") pod \"48b9132d-b7c7-4788-ac38-05ea322a7683\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " Jul 14 19:56:28.352506 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.352359 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-dshm\") pod \"48b9132d-b7c7-4788-ac38-05ea322a7683\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " Jul 14 19:56:28.352506 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.352403 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-tmp-dir\") pod \"48b9132d-b7c7-4788-ac38-05ea322a7683\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " Jul 14 19:56:28.352506 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.352418 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48b9132d-b7c7-4788-ac38-05ea322a7683-tls-certs\") pod \"48b9132d-b7c7-4788-ac38-05ea322a7683\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " Jul 14 19:56:28.352506 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.352437 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-model-cache\") pod \"48b9132d-b7c7-4788-ac38-05ea322a7683\" (UID: \"48b9132d-b7c7-4788-ac38-05ea322a7683\") " Jul 14 19:56:28.352831 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.352578 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-home" (OuterVolumeSpecName: "home") pod "48b9132d-b7c7-4788-ac38-05ea322a7683" (UID: "48b9132d-b7c7-4788-ac38-05ea322a7683"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:56:28.352831 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.352703 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48b9132d-b7c7-4788-ac38-05ea322a7683" (UID: "48b9132d-b7c7-4788-ac38-05ea322a7683"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:56:28.353096 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.353072 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-home\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:56:28.353193 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.353097 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-tmp-dir\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:56:28.353442 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.353404 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-model-cache" (OuterVolumeSpecName: "model-cache") pod "48b9132d-b7c7-4788-ac38-05ea322a7683" (UID: "48b9132d-b7c7-4788-ac38-05ea322a7683"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:56:28.354757 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.354723 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-dshm" (OuterVolumeSpecName: "dshm") pod "48b9132d-b7c7-4788-ac38-05ea322a7683" (UID: "48b9132d-b7c7-4788-ac38-05ea322a7683"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:56:28.354757 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.354741 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b9132d-b7c7-4788-ac38-05ea322a7683-kube-api-access-zfwqq" (OuterVolumeSpecName: "kube-api-access-zfwqq") pod "48b9132d-b7c7-4788-ac38-05ea322a7683" (UID: "48b9132d-b7c7-4788-ac38-05ea322a7683"). InnerVolumeSpecName "kube-api-access-zfwqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:56:28.354891 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.354816 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48b9132d-b7c7-4788-ac38-05ea322a7683-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48b9132d-b7c7-4788-ac38-05ea322a7683" (UID: "48b9132d-b7c7-4788-ac38-05ea322a7683"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:56:28.453668 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.453627 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfwqq\" (UniqueName: \"kubernetes.io/projected/48b9132d-b7c7-4788-ac38-05ea322a7683-kube-api-access-zfwqq\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:56:28.453668 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.453663 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-dshm\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:56:28.453668 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.453677 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48b9132d-b7c7-4788-ac38-05ea322a7683-tls-certs\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:56:28.453898 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.453689 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b9132d-b7c7-4788-ac38-05ea322a7683-model-cache\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:56:28.826491 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.826433 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" podUID="f0716391-d5a6-4410-9b13-4577a018144d" containerName="main" containerID="cri-o://8830072d9e72c21e4541dd197937a924fe20f4fbcffa5beb6b90bd28de177a79" gracePeriod=45 Jul 14 19:56:28.873113 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.873079 2572 generic.go:356] "Generic (PLEG): container finished" podID="48b9132d-b7c7-4788-ac38-05ea322a7683" containerID="6c0cb027331947c96ddec385e6e2ca805bf480f0e2c43a1db6b383c216ce5fba" exitCode=0 Jul 14 19:56:28.873275 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.873128 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" event={"ID":"48b9132d-b7c7-4788-ac38-05ea322a7683","Type":"ContainerDied","Data":"6c0cb027331947c96ddec385e6e2ca805bf480f0e2c43a1db6b383c216ce5fba"} Jul 14 19:56:28.873275 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.873142 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" Jul 14 19:56:28.873275 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.873162 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb" event={"ID":"48b9132d-b7c7-4788-ac38-05ea322a7683","Type":"ContainerDied","Data":"397f58a2f959a6d22897c887a87d615809882c58c1dd97e7efc78cfcc2714b2d"} Jul 14 19:56:28.873275 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.873179 2572 scope.go:117] "RemoveContainer" containerID="6c0cb027331947c96ddec385e6e2ca805bf480f0e2c43a1db6b383c216ce5fba" Jul 14 19:56:28.905723 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.905696 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb"] Jul 14 19:56:28.909302 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.909276 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/auth-test-model-70659c-kserve-679bf587bc-hrvwb"] Jul 14 19:56:28.936674 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.936650 2572 scope.go:117] "RemoveContainer" containerID="6c0cb027331947c96ddec385e6e2ca805bf480f0e2c43a1db6b383c216ce5fba" Jul 14 19:56:28.937009 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:56:28.936990 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0cb027331947c96ddec385e6e2ca805bf480f0e2c43a1db6b383c216ce5fba\": container with ID starting with 6c0cb027331947c96ddec385e6e2ca805bf480f0e2c43a1db6b383c216ce5fba not found: ID does not exist" containerID="6c0cb027331947c96ddec385e6e2ca805bf480f0e2c43a1db6b383c216ce5fba" Jul 14 19:56:28.937106 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:28.937023 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0cb027331947c96ddec385e6e2ca805bf480f0e2c43a1db6b383c216ce5fba"} err="failed to get container status \"6c0cb027331947c96ddec385e6e2ca805bf480f0e2c43a1db6b383c216ce5fba\": rpc error: code = NotFound desc = could not find container \"6c0cb027331947c96ddec385e6e2ca805bf480f0e2c43a1db6b383c216ce5fba\": container with ID starting with 6c0cb027331947c96ddec385e6e2ca805bf480f0e2c43a1db6b383c216ce5fba not found: ID does not exist" Jul 14 19:56:29.087618 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.087559 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:56:29.158989 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.158957 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c4hl\" (UniqueName: \"kubernetes.io/projected/f0716391-d5a6-4410-9b13-4577a018144d-kube-api-access-4c4hl\") pod \"f0716391-d5a6-4410-9b13-4577a018144d\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " Jul 14 19:56:29.158989 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.158999 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-model-cache\") pod \"f0716391-d5a6-4410-9b13-4577a018144d\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " Jul 14 19:56:29.159273 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.159028 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-tmp-dir\") pod \"f0716391-d5a6-4410-9b13-4577a018144d\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " Jul 14 19:56:29.159273 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.159070 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-home\") pod \"f0716391-d5a6-4410-9b13-4577a018144d\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " Jul 14 19:56:29.159273 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.159109 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0716391-d5a6-4410-9b13-4577a018144d-tls-certs\") pod \"f0716391-d5a6-4410-9b13-4577a018144d\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " Jul 14 19:56:29.159273 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.159131 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-dshm\") pod \"f0716391-d5a6-4410-9b13-4577a018144d\" (UID: \"f0716391-d5a6-4410-9b13-4577a018144d\") " Jul 14 19:56:29.159470 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.159293 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0716391-d5a6-4410-9b13-4577a018144d" (UID: "f0716391-d5a6-4410-9b13-4577a018144d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:56:29.159470 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.159312 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0716391-d5a6-4410-9b13-4577a018144d" (UID: "f0716391-d5a6-4410-9b13-4577a018144d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:56:29.159470 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.159385 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-home" (OuterVolumeSpecName: "home") pod "f0716391-d5a6-4410-9b13-4577a018144d" (UID: "f0716391-d5a6-4410-9b13-4577a018144d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:56:29.161276 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.161244 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0716391-d5a6-4410-9b13-4577a018144d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0716391-d5a6-4410-9b13-4577a018144d" (UID: "f0716391-d5a6-4410-9b13-4577a018144d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:56:29.161276 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.161267 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-dshm" (OuterVolumeSpecName: "dshm") pod "f0716391-d5a6-4410-9b13-4577a018144d" (UID: "f0716391-d5a6-4410-9b13-4577a018144d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:56:29.161409 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.161273 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0716391-d5a6-4410-9b13-4577a018144d-kube-api-access-4c4hl" (OuterVolumeSpecName: "kube-api-access-4c4hl") pod "f0716391-d5a6-4410-9b13-4577a018144d" (UID: "f0716391-d5a6-4410-9b13-4577a018144d"). InnerVolumeSpecName "kube-api-access-4c4hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:56:29.187836 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.187806 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48b9132d-b7c7-4788-ac38-05ea322a7683" path="/var/lib/kubelet/pods/48b9132d-b7c7-4788-ac38-05ea322a7683/volumes" Jul 14 19:56:29.259971 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.259933 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-home\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:56:29.259971 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.259964 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0716391-d5a6-4410-9b13-4577a018144d-tls-certs\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:56:29.259971 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.259978 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-dshm\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:56:29.260249 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.259989 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c4hl\" (UniqueName: \"kubernetes.io/projected/f0716391-d5a6-4410-9b13-4577a018144d-kube-api-access-4c4hl\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:56:29.260249 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.260004 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-model-cache\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:56:29.260249 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.260065 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0716391-d5a6-4410-9b13-4577a018144d-tmp-dir\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:56:29.879481 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.879446 2572 generic.go:356] "Generic (PLEG): container finished" podID="f0716391-d5a6-4410-9b13-4577a018144d" containerID="8830072d9e72c21e4541dd197937a924fe20f4fbcffa5beb6b90bd28de177a79" exitCode=0 Jul 14 19:56:29.879685 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.879497 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" Jul 14 19:56:29.879685 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.879543 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" event={"ID":"f0716391-d5a6-4410-9b13-4577a018144d","Type":"ContainerDied","Data":"8830072d9e72c21e4541dd197937a924fe20f4fbcffa5beb6b90bd28de177a79"} Jul 14 19:56:29.879685 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.879585 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w" event={"ID":"f0716391-d5a6-4410-9b13-4577a018144d","Type":"ContainerDied","Data":"d9c6e5dc7b62f2c0e08ad6b7a5fad432367a7975faf47744ea3623b91b4eb10c"} Jul 14 19:56:29.879685 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.879610 2572 scope.go:117] "RemoveContainer" containerID="8830072d9e72c21e4541dd197937a924fe20f4fbcffa5beb6b90bd28de177a79" Jul 14 19:56:29.888647 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.888624 2572 scope.go:117] "RemoveContainer" containerID="8830072d9e72c21e4541dd197937a924fe20f4fbcffa5beb6b90bd28de177a79" Jul 14 19:56:29.888967 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:56:29.888939 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8830072d9e72c21e4541dd197937a924fe20f4fbcffa5beb6b90bd28de177a79\": container with ID starting with 8830072d9e72c21e4541dd197937a924fe20f4fbcffa5beb6b90bd28de177a79 not found: ID does not exist" containerID="8830072d9e72c21e4541dd197937a924fe20f4fbcffa5beb6b90bd28de177a79" Jul 14 19:56:29.889020 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.888977 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8830072d9e72c21e4541dd197937a924fe20f4fbcffa5beb6b90bd28de177a79"} err="failed to get container status \"8830072d9e72c21e4541dd197937a924fe20f4fbcffa5beb6b90bd28de177a79\": rpc error: code = NotFound desc = could not find container \"8830072d9e72c21e4541dd197937a924fe20f4fbcffa5beb6b90bd28de177a79\": container with ID starting with 8830072d9e72c21e4541dd197937a924fe20f4fbcffa5beb6b90bd28de177a79 not found: ID does not exist" Jul 14 19:56:29.896396 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.896360 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w"] Jul 14 19:56:29.898001 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:29.897977 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/auth-test-model-356b53-kserve-5685d5776c-lbc2w"] Jul 14 19:56:30.612441 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:30.612406 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:30.612936 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:30.612560 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:30.625248 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:30.625218 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:30.900676 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:30.900589 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:56:31.189050 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:31.188963 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0716391-d5a6-4410-9b13-4577a018144d" path="/var/lib/kubelet/pods/f0716391-d5a6-4410-9b13-4577a018144d/volumes" Jul 14 19:56:34.227182 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:34.227149 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:56:38.610707 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.610677 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk"] Jul 14 19:56:38.611098 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.610961 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48b9132d-b7c7-4788-ac38-05ea322a7683" containerName="main" Jul 14 19:56:38.611098 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.610972 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="48b9132d-b7c7-4788-ac38-05ea322a7683" containerName="main" Jul 14 19:56:38.611098 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.610980 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0716391-d5a6-4410-9b13-4577a018144d" containerName="main" Jul 14 19:56:38.611098 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.610986 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0716391-d5a6-4410-9b13-4577a018144d" containerName="main" Jul 14 19:56:38.611098 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.611053 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0716391-d5a6-4410-9b13-4577a018144d" containerName="main" Jul 14 19:56:38.611098 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.611062 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="48b9132d-b7c7-4788-ac38-05ea322a7683" containerName="main" Jul 14 19:56:38.615128 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.615108 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.617786 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.617755 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-356b53\"/\"kube-root-ca.crt\"" Jul 14 19:56:38.617907 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.617838 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-356b53\"/\"openshift-service-ca.crt\"" Jul 14 19:56:38.618826 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.618807 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-356b53\"/\"default-dockercfg-d22z5\"" Jul 14 19:56:38.618948 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.618863 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-356b53\"/\"sub-test-model-356b53-kserve-self-signed-certs\"" Jul 14 19:56:38.623812 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.623790 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk"] Jul 14 19:56:38.632264 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.632236 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-dshm\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.632414 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.632270 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-home\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.632414 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.632293 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8b8411-f80e-4774-b57b-08b2ad647708-tls-certs\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.632414 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.632385 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-model-cache\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.632519 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.632432 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vklbq\" (UniqueName: \"kubernetes.io/projected/1f8b8411-f80e-4774-b57b-08b2ad647708-kube-api-access-vklbq\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.632519 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.632477 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-tmp-dir\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.637318 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.637281 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:56:38.733004 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.732964 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-tmp-dir\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.733222 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.733018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-dshm\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.733222 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.733129 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-home\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.733222 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.733179 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8b8411-f80e-4774-b57b-08b2ad647708-tls-certs\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.733400 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.733231 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-model-cache\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.733400 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.733290 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vklbq\" (UniqueName: \"kubernetes.io/projected/1f8b8411-f80e-4774-b57b-08b2ad647708-kube-api-access-vklbq\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.733576 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.733553 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-tmp-dir\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.733647 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.733560 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-home\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.733692 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.733661 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-model-cache\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.735332 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.735312 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-dshm\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.735512 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.735497 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8b8411-f80e-4774-b57b-08b2ad647708-tls-certs\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.741886 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.741858 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vklbq\" (UniqueName: \"kubernetes.io/projected/1f8b8411-f80e-4774-b57b-08b2ad647708-kube-api-access-vklbq\") pod \"sub-test-model-356b53-kserve-588b7cdd79-dc4xk\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:38.924896 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:38.924813 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:39.059669 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:39.059636 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk"] Jul 14 19:56:39.062925 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:56:39.062895 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f8b8411_f80e_4774_b57b_08b2ad647708.slice/crio-29aaa603af051215e7d866fdc33c09db89fd9b0ae4f179e4df769b9e91bfe3f0 WatchSource:0}: Error finding container 29aaa603af051215e7d866fdc33c09db89fd9b0ae4f179e4df769b9e91bfe3f0: Status 404 returned error can't find the container with id 29aaa603af051215e7d866fdc33c09db89fd9b0ae4f179e4df769b9e91bfe3f0 Jul 14 19:56:39.925002 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:39.924967 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" event={"ID":"1f8b8411-f80e-4774-b57b-08b2ad647708","Type":"ContainerStarted","Data":"cf16daac3d080fd355c23a285f8da6e15c4a6869c698020cd069ae0e17b22497"} Jul 14 19:56:39.925002 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:39.925010 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" event={"ID":"1f8b8411-f80e-4774-b57b-08b2ad647708","Type":"ContainerStarted","Data":"29aaa603af051215e7d866fdc33c09db89fd9b0ae4f179e4df769b9e91bfe3f0"} Jul 14 19:56:39.942542 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:39.942487 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" podStartSLOduration=1.9424728359999999 podStartE2EDuration="1.942472836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:56:39.941229719 +0000 UTC m=+2749.306300080" watchObservedRunningTime="2026-07-14 19:56:39.942472836 +0000 UTC m=+2749.307543197" Jul 14 19:56:45.835056 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:45.835016 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:56:48.925743 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:48.925701 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:48.926279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:48.925791 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:48.938133 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:48.938106 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:56:48.973910 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:56:48.973887 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:57:00.841007 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:00.840962 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:57:05.417844 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:05.417764 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:57:16.524676 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:16.524602 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts"] Jul 14 19:57:16.562752 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:16.562712 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:57:17.415493 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:17.415460 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk"] Jul 14 19:57:17.487301 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:17.487249 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:57:22.128110 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:22.128075 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:57:31.544422 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.544379 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" podUID="afa9fa27-e2c1-41d6-a898-ec0b936280bf" containerName="main" containerID="cri-o://056d1a6c9c19f15e26f54bd6f0b488817af39d2795fa83cd35c7fd3b819dbae6" gracePeriod=45 Jul 14 19:57:31.792798 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.792774 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:57:31.875462 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.875428 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-home\") pod \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " Jul 14 19:57:31.875462 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.875471 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa9fa27-e2c1-41d6-a898-ec0b936280bf-tls-certs\") pod \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " Jul 14 19:57:31.875931 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.875498 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-model-cache\") pod \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " Jul 14 19:57:31.875931 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.875523 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-dshm\") pod \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " Jul 14 19:57:31.875931 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.875552 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d5t5\" (UniqueName: \"kubernetes.io/projected/afa9fa27-e2c1-41d6-a898-ec0b936280bf-kube-api-access-6d5t5\") pod \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " Jul 14 19:57:31.875931 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.875573 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-tmp-dir\") pod \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\" (UID: \"afa9fa27-e2c1-41d6-a898-ec0b936280bf\") " Jul 14 19:57:31.875931 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.875764 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-home" (OuterVolumeSpecName: "home") pod "afa9fa27-e2c1-41d6-a898-ec0b936280bf" (UID: "afa9fa27-e2c1-41d6-a898-ec0b936280bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:57:31.875931 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.875848 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "afa9fa27-e2c1-41d6-a898-ec0b936280bf" (UID: "afa9fa27-e2c1-41d6-a898-ec0b936280bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:57:31.876214 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.875924 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "afa9fa27-e2c1-41d6-a898-ec0b936280bf" (UID: "afa9fa27-e2c1-41d6-a898-ec0b936280bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:57:31.877659 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.877637 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa9fa27-e2c1-41d6-a898-ec0b936280bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "afa9fa27-e2c1-41d6-a898-ec0b936280bf" (UID: "afa9fa27-e2c1-41d6-a898-ec0b936280bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:57:31.877748 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.877640 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-dshm" (OuterVolumeSpecName: "dshm") pod "afa9fa27-e2c1-41d6-a898-ec0b936280bf" (UID: "afa9fa27-e2c1-41d6-a898-ec0b936280bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:57:31.877748 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.877716 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa9fa27-e2c1-41d6-a898-ec0b936280bf-kube-api-access-6d5t5" (OuterVolumeSpecName: "kube-api-access-6d5t5") pod "afa9fa27-e2c1-41d6-a898-ec0b936280bf" (UID: "afa9fa27-e2c1-41d6-a898-ec0b936280bf"). InnerVolumeSpecName "kube-api-access-6d5t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:57:31.976112 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.976083 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-home\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:57:31.976112 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.976108 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa9fa27-e2c1-41d6-a898-ec0b936280bf-tls-certs\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:57:31.976112 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.976119 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-model-cache\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:57:31.976112 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.976127 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-dshm\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:57:31.976370 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.976135 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d5t5\" (UniqueName: \"kubernetes.io/projected/afa9fa27-e2c1-41d6-a898-ec0b936280bf-kube-api-access-6d5t5\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:57:31.976370 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:31.976144 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa9fa27-e2c1-41d6-a898-ec0b936280bf-tmp-dir\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:57:32.149313 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.149219 2572 generic.go:356] "Generic (PLEG): container finished" podID="afa9fa27-e2c1-41d6-a898-ec0b936280bf" containerID="056d1a6c9c19f15e26f54bd6f0b488817af39d2795fa83cd35c7fd3b819dbae6" exitCode=0 Jul 14 19:57:32.149313 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.149257 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" Jul 14 19:57:32.149313 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.149287 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" event={"ID":"afa9fa27-e2c1-41d6-a898-ec0b936280bf","Type":"ContainerDied","Data":"056d1a6c9c19f15e26f54bd6f0b488817af39d2795fa83cd35c7fd3b819dbae6"} Jul 14 19:57:32.149527 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.149324 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts" event={"ID":"afa9fa27-e2c1-41d6-a898-ec0b936280bf","Type":"ContainerDied","Data":"d862fa2f75cc82b6aa036ee85cc13e650e0284937849bbc246f8d28b7e3fade9"} Jul 14 19:57:32.149527 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.149344 2572 scope.go:117] "RemoveContainer" containerID="056d1a6c9c19f15e26f54bd6f0b488817af39d2795fa83cd35c7fd3b819dbae6" Jul 14 19:57:32.158001 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.157980 2572 scope.go:117] "RemoveContainer" containerID="056d1a6c9c19f15e26f54bd6f0b488817af39d2795fa83cd35c7fd3b819dbae6" Jul 14 19:57:32.158451 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:57:32.158425 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056d1a6c9c19f15e26f54bd6f0b488817af39d2795fa83cd35c7fd3b819dbae6\": container with ID starting with 056d1a6c9c19f15e26f54bd6f0b488817af39d2795fa83cd35c7fd3b819dbae6 not found: ID does not exist" containerID="056d1a6c9c19f15e26f54bd6f0b488817af39d2795fa83cd35c7fd3b819dbae6" Jul 14 19:57:32.158549 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.158461 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056d1a6c9c19f15e26f54bd6f0b488817af39d2795fa83cd35c7fd3b819dbae6"} err="failed to get container status \"056d1a6c9c19f15e26f54bd6f0b488817af39d2795fa83cd35c7fd3b819dbae6\": rpc error: code = NotFound desc = could not find container \"056d1a6c9c19f15e26f54bd6f0b488817af39d2795fa83cd35c7fd3b819dbae6\": container with ID starting with 056d1a6c9c19f15e26f54bd6f0b488817af39d2795fa83cd35c7fd3b819dbae6 not found: ID does not exist" Jul 14 19:57:32.168486 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.168443 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts"] Jul 14 19:57:32.170512 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.170488 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/sub-test-model-70659c-kserve-7967f95659-j4gts"] Jul 14 19:57:32.434838 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.434747 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" podUID="1f8b8411-f80e-4774-b57b-08b2ad647708" containerName="main" containerID="cri-o://cf16daac3d080fd355c23a285f8da6e15c4a6869c698020cd069ae0e17b22497" gracePeriod=45 Jul 14 19:57:32.679456 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.679432 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:57:32.681450 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.681433 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vklbq\" (UniqueName: \"kubernetes.io/projected/1f8b8411-f80e-4774-b57b-08b2ad647708-kube-api-access-vklbq\") pod \"1f8b8411-f80e-4774-b57b-08b2ad647708\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " Jul 14 19:57:32.681539 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.681493 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-tmp-dir\") pod \"1f8b8411-f80e-4774-b57b-08b2ad647708\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " Jul 14 19:57:32.681539 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.681520 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-dshm\") pod \"1f8b8411-f80e-4774-b57b-08b2ad647708\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " Jul 14 19:57:32.681688 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.681671 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-model-cache\") pod \"1f8b8411-f80e-4774-b57b-08b2ad647708\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " Jul 14 19:57:32.681747 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.681705 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-home\") pod \"1f8b8411-f80e-4774-b57b-08b2ad647708\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " Jul 14 19:57:32.681747 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.681742 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8b8411-f80e-4774-b57b-08b2ad647708-tls-certs\") pod \"1f8b8411-f80e-4774-b57b-08b2ad647708\" (UID: \"1f8b8411-f80e-4774-b57b-08b2ad647708\") " Jul 14 19:57:32.681865 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.681778 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f8b8411-f80e-4774-b57b-08b2ad647708" (UID: "1f8b8411-f80e-4774-b57b-08b2ad647708"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:57:32.681917 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.681898 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-home" (OuterVolumeSpecName: "home") pod "1f8b8411-f80e-4774-b57b-08b2ad647708" (UID: "1f8b8411-f80e-4774-b57b-08b2ad647708"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:57:32.681971 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.681919 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f8b8411-f80e-4774-b57b-08b2ad647708" (UID: "1f8b8411-f80e-4774-b57b-08b2ad647708"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:57:32.681971 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.681966 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-home\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:57:32.682100 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.681979 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-tmp-dir\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:57:32.683615 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.683591 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-dshm" (OuterVolumeSpecName: "dshm") pod "1f8b8411-f80e-4774-b57b-08b2ad647708" (UID: "1f8b8411-f80e-4774-b57b-08b2ad647708"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:57:32.683709 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.683623 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8b8411-f80e-4774-b57b-08b2ad647708-kube-api-access-vklbq" (OuterVolumeSpecName: "kube-api-access-vklbq") pod "1f8b8411-f80e-4774-b57b-08b2ad647708" (UID: "1f8b8411-f80e-4774-b57b-08b2ad647708"). InnerVolumeSpecName "kube-api-access-vklbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:57:32.683774 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.683752 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f8b8411-f80e-4774-b57b-08b2ad647708-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f8b8411-f80e-4774-b57b-08b2ad647708" (UID: "1f8b8411-f80e-4774-b57b-08b2ad647708"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:57:32.782379 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.782344 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-dshm\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:57:32.782379 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.782371 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f8b8411-f80e-4774-b57b-08b2ad647708-model-cache\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:57:32.782379 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.782380 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8b8411-f80e-4774-b57b-08b2ad647708-tls-certs\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:57:32.782379 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:32.782389 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vklbq\" (UniqueName: \"kubernetes.io/projected/1f8b8411-f80e-4774-b57b-08b2ad647708-kube-api-access-vklbq\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:57:33.155382 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:33.155352 2572 generic.go:356] "Generic (PLEG): container finished" podID="1f8b8411-f80e-4774-b57b-08b2ad647708" containerID="cf16daac3d080fd355c23a285f8da6e15c4a6869c698020cd069ae0e17b22497" exitCode=0 Jul 14 19:57:33.155605 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:33.155409 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" Jul 14 19:57:33.155605 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:33.155443 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" event={"ID":"1f8b8411-f80e-4774-b57b-08b2ad647708","Type":"ContainerDied","Data":"cf16daac3d080fd355c23a285f8da6e15c4a6869c698020cd069ae0e17b22497"} Jul 14 19:57:33.155605 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:33.155485 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk" event={"ID":"1f8b8411-f80e-4774-b57b-08b2ad647708","Type":"ContainerDied","Data":"29aaa603af051215e7d866fdc33c09db89fd9b0ae4f179e4df769b9e91bfe3f0"} Jul 14 19:57:33.155605 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:33.155504 2572 scope.go:117] "RemoveContainer" containerID="cf16daac3d080fd355c23a285f8da6e15c4a6869c698020cd069ae0e17b22497" Jul 14 19:57:33.164037 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:33.164018 2572 scope.go:117] "RemoveContainer" containerID="cf16daac3d080fd355c23a285f8da6e15c4a6869c698020cd069ae0e17b22497" Jul 14 19:57:33.164420 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:57:33.164402 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf16daac3d080fd355c23a285f8da6e15c4a6869c698020cd069ae0e17b22497\": container with ID starting with cf16daac3d080fd355c23a285f8da6e15c4a6869c698020cd069ae0e17b22497 not found: ID does not exist" containerID="cf16daac3d080fd355c23a285f8da6e15c4a6869c698020cd069ae0e17b22497" Jul 14 19:57:33.164494 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:33.164430 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf16daac3d080fd355c23a285f8da6e15c4a6869c698020cd069ae0e17b22497"} err="failed to get container status \"cf16daac3d080fd355c23a285f8da6e15c4a6869c698020cd069ae0e17b22497\": rpc error: code = NotFound desc = could not find container \"cf16daac3d080fd355c23a285f8da6e15c4a6869c698020cd069ae0e17b22497\": container with ID starting with cf16daac3d080fd355c23a285f8da6e15c4a6869c698020cd069ae0e17b22497 not found: ID does not exist" Jul 14 19:57:33.174756 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:33.174731 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk"] Jul 14 19:57:33.178766 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:33.178744 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/sub-test-model-356b53-kserve-588b7cdd79-dc4xk"] Jul 14 19:57:33.189498 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:33.189473 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f8b8411-f80e-4774-b57b-08b2ad647708" path="/var/lib/kubelet/pods/1f8b8411-f80e-4774-b57b-08b2ad647708/volumes" Jul 14 19:57:33.189844 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:33.189831 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afa9fa27-e2c1-41d6-a898-ec0b936280bf" path="/var/lib/kubelet/pods/afa9fa27-e2c1-41d6-a898-ec0b936280bf/volumes" Jul 14 19:57:35.707463 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.707431 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2"] Jul 14 19:57:35.707928 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.707851 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afa9fa27-e2c1-41d6-a898-ec0b936280bf" containerName="main" Jul 14 19:57:35.707928 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.707869 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="afa9fa27-e2c1-41d6-a898-ec0b936280bf" containerName="main" Jul 14 19:57:35.707928 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.707884 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8b8411-f80e-4774-b57b-08b2ad647708" containerName="main" Jul 14 19:57:35.707928 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.707892 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8b8411-f80e-4774-b57b-08b2ad647708" containerName="main" Jul 14 19:57:35.708169 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.707980 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8b8411-f80e-4774-b57b-08b2ad647708" containerName="main" Jul 14 19:57:35.708169 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.707993 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="afa9fa27-e2c1-41d6-a898-ec0b936280bf" containerName="main" Jul 14 19:57:35.712283 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.712264 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.715105 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.715084 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-70659c\"/\"openshift-service-ca.crt\"" Jul 14 19:57:35.715214 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.715084 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-70659c\"/\"kube-root-ca.crt\"" Jul 14 19:57:35.716364 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.716344 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-70659c\"/\"rate-test-model-70659c-kserve-self-signed-certs\"" Jul 14 19:57:35.716364 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.716348 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-70659c\"/\"default-dockercfg-s5ftl\"" Jul 14 19:57:35.722476 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.722457 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2"] Jul 14 19:57:35.743002 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.742972 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:57:35.808279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.808224 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-tmp-dir\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.808279 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.808280 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628eae60-3d50-4d1e-966d-4d0a571d1ead-tls-certs\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.808507 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.808383 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-dshm\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.808507 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.808413 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-home\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.808507 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.808447 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-model-cache\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.808606 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.808499 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvqg6\" (UniqueName: \"kubernetes.io/projected/628eae60-3d50-4d1e-966d-4d0a571d1ead-kube-api-access-dvqg6\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.909576 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.909538 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-tmp-dir\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.909576 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.909575 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628eae60-3d50-4d1e-966d-4d0a571d1ead-tls-certs\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.909828 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.909625 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-dshm\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.909828 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.909641 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-home\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.909828 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.909666 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-model-cache\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.909828 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.909693 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvqg6\" (UniqueName: \"kubernetes.io/projected/628eae60-3d50-4d1e-966d-4d0a571d1ead-kube-api-access-dvqg6\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.910070 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.909990 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-tmp-dir\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.910144 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.910093 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-home\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.910144 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.910126 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-model-cache\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.911871 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.911851 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-dshm\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.912154 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.912135 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628eae60-3d50-4d1e-966d-4d0a571d1ead-tls-certs\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:35.917058 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:35.917022 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvqg6\" (UniqueName: \"kubernetes.io/projected/628eae60-3d50-4d1e-966d-4d0a571d1ead-kube-api-access-dvqg6\") pod \"rate-test-model-70659c-kserve-6884658587-5nmc2\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:36.021794 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:36.021687 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:36.152088 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:36.152029 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2"] Jul 14 19:57:36.154289 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:57:36.154250 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod628eae60_3d50_4d1e_966d_4d0a571d1ead.slice/crio-b3380515bdebee7484bbcf047f8ef6cbe5d84e5e4a59e57b109070a988ef5813 WatchSource:0}: Error finding container b3380515bdebee7484bbcf047f8ef6cbe5d84e5e4a59e57b109070a988ef5813: Status 404 returned error can't find the container with id b3380515bdebee7484bbcf047f8ef6cbe5d84e5e4a59e57b109070a988ef5813 Jul 14 19:57:36.172174 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:36.172147 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" event={"ID":"628eae60-3d50-4d1e-966d-4d0a571d1ead","Type":"ContainerStarted","Data":"b3380515bdebee7484bbcf047f8ef6cbe5d84e5e4a59e57b109070a988ef5813"} Jul 14 19:57:37.180022 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:37.179986 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" event={"ID":"628eae60-3d50-4d1e-966d-4d0a571d1ead","Type":"ContainerStarted","Data":"5da7bcee5278b1e65ffdb99702f2729b61d2337e027633c03bd681d0d1fa355f"} Jul 14 19:57:37.196485 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:37.196436 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" podStartSLOduration=2.196419646 podStartE2EDuration="2.196419646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:57:37.194992148 +0000 UTC m=+2806.560062511" watchObservedRunningTime="2026-07-14 19:57:37.196419646 +0000 UTC m=+2806.561490007" Jul 14 19:57:40.029534 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:40.029492 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:57:46.021993 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:46.021948 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:46.022436 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:46.022011 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:46.034508 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:46.034480 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:57:46.231549 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:57:46.231519 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:58:09.303365 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.303331 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb"] Jul 14 19:58:09.306950 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.306926 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.310770 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.310744 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-356b53\"/\"kube-root-ca.crt\"" Jul 14 19:58:09.310914 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.310749 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-356b53\"/\"rate-test-model-356b53-kserve-self-signed-certs\"" Jul 14 19:58:09.310914 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.310749 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-356b53\"/\"default-dockercfg-d22z5\"" Jul 14 19:58:09.310914 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.310751 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-356b53\"/\"openshift-service-ca.crt\"" Jul 14 19:58:09.315433 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.315413 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb"] Jul 14 19:58:09.396616 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.396579 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c888db-8d47-4771-8439-c7a7e5893394-tls-certs\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.396800 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.396627 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-home\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.396800 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.396646 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8srd\" (UniqueName: \"kubernetes.io/projected/88c888db-8d47-4771-8439-c7a7e5893394-kube-api-access-c8srd\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.396800 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.396719 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-tmp-dir\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.396800 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.396756 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-dshm\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.396800 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.396775 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-model-cache\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.497352 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.497312 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-dshm\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.497352 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.497350 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-model-cache\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.497554 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.497481 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c888db-8d47-4771-8439-c7a7e5893394-tls-certs\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.497554 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.497536 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-home\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.497554 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.497553 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8srd\" (UniqueName: \"kubernetes.io/projected/88c888db-8d47-4771-8439-c7a7e5893394-kube-api-access-c8srd\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.497710 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.497596 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-tmp-dir\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.497824 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.497803 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-model-cache\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.497892 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.497855 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-home\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.497939 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.497914 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-tmp-dir\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.499755 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.499731 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-dshm\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.499890 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.499875 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c888db-8d47-4771-8439-c7a7e5893394-tls-certs\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.505996 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.505972 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8srd\" (UniqueName: \"kubernetes.io/projected/88c888db-8d47-4771-8439-c7a7e5893394-kube-api-access-c8srd\") pod \"rate-test-model-356b53-kserve-6d85d66b9f-n6vtb\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.616760 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.616717 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:09.745486 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:09.745459 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb"] Jul 14 19:58:09.748084 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:58:09.748016 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88c888db_8d47_4771_8439_c7a7e5893394.slice/crio-4605d638e2acfa4f19579d5fbf9feb7668fee5ebee5c1b72368ab2e9061c2bb2 WatchSource:0}: Error finding container 4605d638e2acfa4f19579d5fbf9feb7668fee5ebee5c1b72368ab2e9061c2bb2: Status 404 returned error can't find the container with id 4605d638e2acfa4f19579d5fbf9feb7668fee5ebee5c1b72368ab2e9061c2bb2 Jul 14 19:58:10.314943 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:10.314908 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" event={"ID":"88c888db-8d47-4771-8439-c7a7e5893394","Type":"ContainerStarted","Data":"d7d29b85af91bb28e0614f72c8f48175a2f4dbbd5e630c766162605c80119ad5"} Jul 14 19:58:10.314943 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:10.314949 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" event={"ID":"88c888db-8d47-4771-8439-c7a7e5893394","Type":"ContainerStarted","Data":"4605d638e2acfa4f19579d5fbf9feb7668fee5ebee5c1b72368ab2e9061c2bb2"} Jul 14 19:58:10.331527 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:10.331472 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" podStartSLOduration=1.33145446 podStartE2EDuration="1.33145446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:58:10.330561453 +0000 UTC m=+2839.695632036" watchObservedRunningTime="2026-07-14 19:58:10.33145446 +0000 UTC m=+2839.696524820" Jul 14 19:58:10.435355 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:10.435326 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:58:13.829768 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:13.829731 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:58:19.617467 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:19.617423 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:19.617892 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:19.617516 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:19.630067 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:19.630026 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:20.370944 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:20.370917 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:58:35.329750 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:35.329713 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:58:50.022899 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:50.022817 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:58:50.123582 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:50.123551 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:58:50.433588 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:50.433554 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:58:57.436903 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:57.436868 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2"] Jul 14 19:58:57.460446 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:57.460410 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:58:58.536818 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:58.536788 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb"] Jul 14 19:58:58.582385 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:58.582357 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:58:59.009457 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:59.009424 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:58:59.114448 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:58:59.114413 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:02.755117 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:02.755078 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7431-140405581687344"] Jul 14 19:59:02.758348 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:02.758325 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581687344" Jul 14 19:59:02.760698 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:02.760675 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zrwrl\"" Jul 14 19:59:02.764380 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:02.764359 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140405581687344"] Jul 14 19:59:02.839862 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:02.839828 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9rgs\" (UniqueName: \"kubernetes.io/projected/9961e371-2c45-47cb-8a78-dd10afd338f2-kube-api-access-n9rgs\") pod \"test-curl-7431-140405581687344\" (UID: \"9961e371-2c45-47cb-8a78-dd10afd338f2\") " pod="opendatahub/test-curl-7431-140405581687344" Jul 14 19:59:02.940374 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:02.940332 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9rgs\" (UniqueName: \"kubernetes.io/projected/9961e371-2c45-47cb-8a78-dd10afd338f2-kube-api-access-n9rgs\") pod \"test-curl-7431-140405581687344\" (UID: \"9961e371-2c45-47cb-8a78-dd10afd338f2\") " pod="opendatahub/test-curl-7431-140405581687344" Jul 14 19:59:02.948420 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:02.948388 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9rgs\" (UniqueName: \"kubernetes.io/projected/9961e371-2c45-47cb-8a78-dd10afd338f2-kube-api-access-n9rgs\") pod \"test-curl-7431-140405581687344\" (UID: \"9961e371-2c45-47cb-8a78-dd10afd338f2\") " pod="opendatahub/test-curl-7431-140405581687344" Jul 14 19:59:03.067438 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:03.067344 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581687344" Jul 14 19:59:03.191735 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:03.191712 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140405581687344"] Jul 14 19:59:03.194013 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:59:03.193987 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9961e371_2c45_47cb_8a78_dd10afd338f2.slice/crio-d2c7469d2a36d0020df943ec383c23c099207d477738d1dfb4ad66389fd80299 WatchSource:0}: Error finding container d2c7469d2a36d0020df943ec383c23c099207d477738d1dfb4ad66389fd80299: Status 404 returned error can't find the container with id d2c7469d2a36d0020df943ec383c23c099207d477738d1dfb4ad66389fd80299 Jul 14 19:59:03.195251 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:03.195231 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:59:03.532075 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:03.532013 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140405581687344" event={"ID":"9961e371-2c45-47cb-8a78-dd10afd338f2","Type":"ContainerStarted","Data":"d2c7469d2a36d0020df943ec383c23c099207d477738d1dfb4ad66389fd80299"} Jul 14 19:59:04.537716 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:04.537682 2572 generic.go:356] "Generic (PLEG): container finished" podID="9961e371-2c45-47cb-8a78-dd10afd338f2" containerID="cab5faf793abd0982f3a0391c3bfae09651cccae97b61df61a1f513a914378ba" exitCode=0 Jul 14 19:59:04.538127 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:04.537754 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140405581687344" event={"ID":"9961e371-2c45-47cb-8a78-dd10afd338f2","Type":"ContainerDied","Data":"cab5faf793abd0982f3a0391c3bfae09651cccae97b61df61a1f513a914378ba"} Jul 14 19:59:05.323772 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:05.323730 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:59:05.681753 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:05.681726 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581687344" Jul 14 19:59:05.701663 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:05.701633 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7431-140405581687344_9961e371-2c45-47cb-8a78-dd10afd338f2/test-curl-7431-140405581687344/0.log" Jul 14 19:59:05.724829 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:05.724799 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7431-140405581687344"] Jul 14 19:59:05.729204 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:05.729178 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7431-140405581687344"] Jul 14 19:59:05.762097 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:05.762075 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9rgs\" (UniqueName: \"kubernetes.io/projected/9961e371-2c45-47cb-8a78-dd10afd338f2-kube-api-access-n9rgs\") pod \"9961e371-2c45-47cb-8a78-dd10afd338f2\" (UID: \"9961e371-2c45-47cb-8a78-dd10afd338f2\") " Jul 14 19:59:05.764162 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:05.764138 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9961e371-2c45-47cb-8a78-dd10afd338f2-kube-api-access-n9rgs" (OuterVolumeSpecName: "kube-api-access-n9rgs") pod "9961e371-2c45-47cb-8a78-dd10afd338f2" (UID: "9961e371-2c45-47cb-8a78-dd10afd338f2"). InnerVolumeSpecName "kube-api-access-n9rgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:59:05.862835 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:05.862749 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9rgs\" (UniqueName: \"kubernetes.io/projected/9961e371-2c45-47cb-8a78-dd10afd338f2-kube-api-access-n9rgs\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:06.548777 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:06.548748 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581687344" Jul 14 19:59:06.548777 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:06.548772 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2c7469d2a36d0020df943ec383c23c099207d477738d1dfb4ad66389fd80299" Jul 14 19:59:06.558497 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:59:06.558469 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7431-140405581687344\" is forbidden: User \"system:node:ip-10-0-138-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-70.ec2.internal' and this object" podUID="9961e371-2c45-47cb-8a78-dd10afd338f2" pod="opendatahub/test-curl-7431-140405581687344" Jul 14 19:59:07.188398 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:07.188356 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9961e371-2c45-47cb-8a78-dd10afd338f2" path="/var/lib/kubelet/pods/9961e371-2c45-47cb-8a78-dd10afd338f2/volumes" Jul 14 19:59:08.782086 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:08.782031 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7431-140405581852720"] Jul 14 19:59:08.782667 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:08.782342 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9961e371-2c45-47cb-8a78-dd10afd338f2" containerName="test-curl-7431-140405581687344" Jul 14 19:59:08.782667 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:08.782353 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9961e371-2c45-47cb-8a78-dd10afd338f2" containerName="test-curl-7431-140405581687344" Jul 14 19:59:08.782667 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:08.782400 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9961e371-2c45-47cb-8a78-dd10afd338f2" containerName="test-curl-7431-140405581687344" Jul 14 19:59:08.785386 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:08.785365 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:08.787938 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:08.787913 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zrwrl\"" Jul 14 19:59:08.791178 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:08.791146 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140405581852720"] Jul 14 19:59:08.887692 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:08.887653 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p62dg\" (UniqueName: \"kubernetes.io/projected/c425096a-1ff8-4ae7-9f40-f01af1c675ce-kube-api-access-p62dg\") pod \"test-curl-7431-140405581852720\" (UID: \"c425096a-1ff8-4ae7-9f40-f01af1c675ce\") " pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:08.988465 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:08.988429 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p62dg\" (UniqueName: \"kubernetes.io/projected/c425096a-1ff8-4ae7-9f40-f01af1c675ce-kube-api-access-p62dg\") pod \"test-curl-7431-140405581852720\" (UID: \"c425096a-1ff8-4ae7-9f40-f01af1c675ce\") " pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:08.997152 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:08.997118 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p62dg\" (UniqueName: \"kubernetes.io/projected/c425096a-1ff8-4ae7-9f40-f01af1c675ce-kube-api-access-p62dg\") pod \"test-curl-7431-140405581852720\" (UID: \"c425096a-1ff8-4ae7-9f40-f01af1c675ce\") " pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:09.096535 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:09.096503 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:09.228102 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:09.228067 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140405581852720"] Jul 14 19:59:09.230948 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:59:09.230919 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc425096a_1ff8_4ae7_9f40_f01af1c675ce.slice/crio-1828226a2246183323d8c1484a0899a301710d32d6f4175220e8ac31fbb4b076 WatchSource:0}: Error finding container 1828226a2246183323d8c1484a0899a301710d32d6f4175220e8ac31fbb4b076: Status 404 returned error can't find the container with id 1828226a2246183323d8c1484a0899a301710d32d6f4175220e8ac31fbb4b076 Jul 14 19:59:09.563148 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:09.563123 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140405581852720" event={"ID":"c425096a-1ff8-4ae7-9f40-f01af1c675ce","Type":"ContainerStarted","Data":"1828226a2246183323d8c1484a0899a301710d32d6f4175220e8ac31fbb4b076"} Jul 14 19:59:10.568822 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:10.568788 2572 generic.go:356] "Generic (PLEG): container finished" podID="c425096a-1ff8-4ae7-9f40-f01af1c675ce" containerID="4576f1e545015515b14c92525f6bb592aec8612bbd2078dc58d2a965d2bccfc3" exitCode=0 Jul 14 19:59:10.569260 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:10.568877 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140405581852720" event={"ID":"c425096a-1ff8-4ae7-9f40-f01af1c675ce","Type":"ContainerDied","Data":"4576f1e545015515b14c92525f6bb592aec8612bbd2078dc58d2a965d2bccfc3"} Jul 14 19:59:11.705634 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:11.705605 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:11.725296 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:11.725275 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7431-140405581852720_c425096a-1ff8-4ae7-9f40-f01af1c675ce/test-curl-7431-140405581852720/0.log" Jul 14 19:59:11.749165 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:11.749134 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7431-140405581852720"] Jul 14 19:59:11.752812 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:11.752787 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7431-140405581852720"] Jul 14 19:59:11.814477 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:11.814445 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p62dg\" (UniqueName: \"kubernetes.io/projected/c425096a-1ff8-4ae7-9f40-f01af1c675ce-kube-api-access-p62dg\") pod \"c425096a-1ff8-4ae7-9f40-f01af1c675ce\" (UID: \"c425096a-1ff8-4ae7-9f40-f01af1c675ce\") " Jul 14 19:59:11.816505 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:11.816483 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c425096a-1ff8-4ae7-9f40-f01af1c675ce-kube-api-access-p62dg" (OuterVolumeSpecName: "kube-api-access-p62dg") pod "c425096a-1ff8-4ae7-9f40-f01af1c675ce" (UID: "c425096a-1ff8-4ae7-9f40-f01af1c675ce"). InnerVolumeSpecName "kube-api-access-p62dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:59:11.874452 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:11.874419 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7431-140405581853728"] Jul 14 19:59:11.874740 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:11.874728 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c425096a-1ff8-4ae7-9f40-f01af1c675ce" containerName="test-curl-7431-140405581852720" Jul 14 19:59:11.874784 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:11.874741 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c425096a-1ff8-4ae7-9f40-f01af1c675ce" containerName="test-curl-7431-140405581852720" Jul 14 19:59:11.874818 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:11.874813 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c425096a-1ff8-4ae7-9f40-f01af1c675ce" containerName="test-curl-7431-140405581852720" Jul 14 19:59:11.877654 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:11.877638 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:11.883296 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:11.883268 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140405581853728"] Jul 14 19:59:11.916142 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:11.916102 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p62dg\" (UniqueName: \"kubernetes.io/projected/c425096a-1ff8-4ae7-9f40-f01af1c675ce-kube-api-access-p62dg\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:12.016848 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.016781 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jmv\" (UniqueName: \"kubernetes.io/projected/4ed2b1ce-c446-40b1-95de-24c11fe8588a-kube-api-access-s7jmv\") pod \"test-curl-7431-140405581853728\" (UID: \"4ed2b1ce-c446-40b1-95de-24c11fe8588a\") " pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:12.118079 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.118022 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jmv\" (UniqueName: \"kubernetes.io/projected/4ed2b1ce-c446-40b1-95de-24c11fe8588a-kube-api-access-s7jmv\") pod \"test-curl-7431-140405581853728\" (UID: \"4ed2b1ce-c446-40b1-95de-24c11fe8588a\") " pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:12.127459 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.127405 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jmv\" (UniqueName: \"kubernetes.io/projected/4ed2b1ce-c446-40b1-95de-24c11fe8588a-kube-api-access-s7jmv\") pod \"test-curl-7431-140405581853728\" (UID: \"4ed2b1ce-c446-40b1-95de-24c11fe8588a\") " pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:12.187519 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.187479 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:12.314357 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.314327 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140405581853728"] Jul 14 19:59:12.457916 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.457815 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" podUID="628eae60-3d50-4d1e-966d-4d0a571d1ead" containerName="main" containerID="cri-o://5da7bcee5278b1e65ffdb99702f2729b61d2337e027633c03bd681d0d1fa355f" gracePeriod=45 Jul 14 19:59:12.579462 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.579432 2572 generic.go:356] "Generic (PLEG): container finished" podID="628eae60-3d50-4d1e-966d-4d0a571d1ead" containerID="5da7bcee5278b1e65ffdb99702f2729b61d2337e027633c03bd681d0d1fa355f" exitCode=0 Jul 14 19:59:12.579627 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.579506 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" event={"ID":"628eae60-3d50-4d1e-966d-4d0a571d1ead","Type":"ContainerDied","Data":"5da7bcee5278b1e65ffdb99702f2729b61d2337e027633c03bd681d0d1fa355f"} Jul 14 19:59:12.580860 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.580832 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140405581853728" event={"ID":"4ed2b1ce-c446-40b1-95de-24c11fe8588a","Type":"ContainerStarted","Data":"f725c94ce028db74322355b93292b7beb1f0405ec016f000375ad0bf9b5c3c75"} Jul 14 19:59:12.582168 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.582148 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1828226a2246183323d8c1484a0899a301710d32d6f4175220e8ac31fbb4b076" Jul 14 19:59:12.582168 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.582165 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:12.592462 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:59:12.592431 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7431-140405581852720\" is forbidden: User \"system:node:ip-10-0-138-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-70.ec2.internal' and this object" podUID="c425096a-1ff8-4ae7-9f40-f01af1c675ce" pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:12.751245 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.751180 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:59:12.753514 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:59:12.753480 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7431-140405581852720\" is forbidden: User \"system:node:ip-10-0-138-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-70.ec2.internal' and this object" podUID="c425096a-1ff8-4ae7-9f40-f01af1c675ce" pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:12.924755 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.924723 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628eae60-3d50-4d1e-966d-4d0a571d1ead-tls-certs\") pod \"628eae60-3d50-4d1e-966d-4d0a571d1ead\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " Jul 14 19:59:12.924755 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.924759 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-dshm\") pod \"628eae60-3d50-4d1e-966d-4d0a571d1ead\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " Jul 14 19:59:12.925034 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.924779 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-model-cache\") pod \"628eae60-3d50-4d1e-966d-4d0a571d1ead\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " Jul 14 19:59:12.925034 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.924796 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-home\") pod \"628eae60-3d50-4d1e-966d-4d0a571d1ead\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " Jul 14 19:59:12.925034 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.924996 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-tmp-dir\") pod \"628eae60-3d50-4d1e-966d-4d0a571d1ead\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " Jul 14 19:59:12.925226 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.925036 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvqg6\" (UniqueName: \"kubernetes.io/projected/628eae60-3d50-4d1e-966d-4d0a571d1ead-kube-api-access-dvqg6\") pod \"628eae60-3d50-4d1e-966d-4d0a571d1ead\" (UID: \"628eae60-3d50-4d1e-966d-4d0a571d1ead\") " Jul 14 19:59:12.925226 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.925078 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-home" (OuterVolumeSpecName: "home") pod "628eae60-3d50-4d1e-966d-4d0a571d1ead" (UID: "628eae60-3d50-4d1e-966d-4d0a571d1ead"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:59:12.925226 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.925098 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-model-cache" (OuterVolumeSpecName: "model-cache") pod "628eae60-3d50-4d1e-966d-4d0a571d1ead" (UID: "628eae60-3d50-4d1e-966d-4d0a571d1ead"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:59:12.925326 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.925230 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "628eae60-3d50-4d1e-966d-4d0a571d1ead" (UID: "628eae60-3d50-4d1e-966d-4d0a571d1ead"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:59:12.925365 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.925332 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-tmp-dir\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:12.925365 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.925352 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-model-cache\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:12.925432 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.925368 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-home\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:12.926957 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.926925 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-dshm" (OuterVolumeSpecName: "dshm") pod "628eae60-3d50-4d1e-966d-4d0a571d1ead" (UID: "628eae60-3d50-4d1e-966d-4d0a571d1ead"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:59:12.927103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.927010 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/628eae60-3d50-4d1e-966d-4d0a571d1ead-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "628eae60-3d50-4d1e-966d-4d0a571d1ead" (UID: "628eae60-3d50-4d1e-966d-4d0a571d1ead"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:59:12.927103 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:12.927033 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628eae60-3d50-4d1e-966d-4d0a571d1ead-kube-api-access-dvqg6" (OuterVolumeSpecName: "kube-api-access-dvqg6") pod "628eae60-3d50-4d1e-966d-4d0a571d1ead" (UID: "628eae60-3d50-4d1e-966d-4d0a571d1ead"). InnerVolumeSpecName "kube-api-access-dvqg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:59:13.026670 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.026576 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvqg6\" (UniqueName: \"kubernetes.io/projected/628eae60-3d50-4d1e-966d-4d0a571d1ead-kube-api-access-dvqg6\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:13.026670 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.026619 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628eae60-3d50-4d1e-966d-4d0a571d1ead-tls-certs\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:13.026670 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.026630 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628eae60-3d50-4d1e-966d-4d0a571d1ead-dshm\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:13.188707 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.188674 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c425096a-1ff8-4ae7-9f40-f01af1c675ce" path="/var/lib/kubelet/pods/c425096a-1ff8-4ae7-9f40-f01af1c675ce/volumes" Jul 14 19:59:13.556502 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.556462 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" podUID="88c888db-8d47-4771-8439-c7a7e5893394" containerName="main" containerID="cri-o://d7d29b85af91bb28e0614f72c8f48175a2f4dbbd5e630c766162605c80119ad5" gracePeriod=45 Jul 14 19:59:13.587808 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.587776 2572 generic.go:356] "Generic (PLEG): container finished" podID="4ed2b1ce-c446-40b1-95de-24c11fe8588a" containerID="764746a9dbee63732fc72b2fde975b4d42d5701ba3a0dfdd8e33c6f06641cd07" exitCode=0 Jul 14 19:59:13.587934 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.587861 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140405581853728" event={"ID":"4ed2b1ce-c446-40b1-95de-24c11fe8588a","Type":"ContainerDied","Data":"764746a9dbee63732fc72b2fde975b4d42d5701ba3a0dfdd8e33c6f06641cd07"} Jul 14 19:59:13.589208 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.589185 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" event={"ID":"628eae60-3d50-4d1e-966d-4d0a571d1ead","Type":"ContainerDied","Data":"b3380515bdebee7484bbcf047f8ef6cbe5d84e5e4a59e57b109070a988ef5813"} Jul 14 19:59:13.589297 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.589210 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2" Jul 14 19:59:13.589297 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.589228 2572 scope.go:117] "RemoveContainer" containerID="5da7bcee5278b1e65ffdb99702f2729b61d2337e027633c03bd681d0d1fa355f" Jul 14 19:59:13.636324 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.636299 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2"] Jul 14 19:59:13.639634 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.639613 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-70659c/rate-test-model-70659c-kserve-6884658587-5nmc2"] Jul 14 19:59:13.802933 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.802912 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:59:13.934917 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.934883 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-home\") pod \"88c888db-8d47-4771-8439-c7a7e5893394\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " Jul 14 19:59:13.934917 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.934923 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8srd\" (UniqueName: \"kubernetes.io/projected/88c888db-8d47-4771-8439-c7a7e5893394-kube-api-access-c8srd\") pod \"88c888db-8d47-4771-8439-c7a7e5893394\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " Jul 14 19:59:13.935194 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.934949 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-tmp-dir\") pod \"88c888db-8d47-4771-8439-c7a7e5893394\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " Jul 14 19:59:13.935194 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.934966 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-dshm\") pod \"88c888db-8d47-4771-8439-c7a7e5893394\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " Jul 14 19:59:13.935194 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.934990 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c888db-8d47-4771-8439-c7a7e5893394-tls-certs\") pod \"88c888db-8d47-4771-8439-c7a7e5893394\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " Jul 14 19:59:13.935194 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.935091 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-model-cache\") pod \"88c888db-8d47-4771-8439-c7a7e5893394\" (UID: \"88c888db-8d47-4771-8439-c7a7e5893394\") " Jul 14 19:59:13.935410 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.935314 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88c888db-8d47-4771-8439-c7a7e5893394" (UID: "88c888db-8d47-4771-8439-c7a7e5893394"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:59:13.935410 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.935326 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-home" (OuterVolumeSpecName: "home") pod "88c888db-8d47-4771-8439-c7a7e5893394" (UID: "88c888db-8d47-4771-8439-c7a7e5893394"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:59:13.935410 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.935386 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-model-cache" (OuterVolumeSpecName: "model-cache") pod "88c888db-8d47-4771-8439-c7a7e5893394" (UID: "88c888db-8d47-4771-8439-c7a7e5893394"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:59:13.937235 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.937212 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c888db-8d47-4771-8439-c7a7e5893394-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88c888db-8d47-4771-8439-c7a7e5893394" (UID: "88c888db-8d47-4771-8439-c7a7e5893394"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:59:13.937345 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.937260 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-dshm" (OuterVolumeSpecName: "dshm") pod "88c888db-8d47-4771-8439-c7a7e5893394" (UID: "88c888db-8d47-4771-8439-c7a7e5893394"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:59:13.937345 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:13.937265 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c888db-8d47-4771-8439-c7a7e5893394-kube-api-access-c8srd" (OuterVolumeSpecName: "kube-api-access-c8srd") pod "88c888db-8d47-4771-8439-c7a7e5893394" (UID: "88c888db-8d47-4771-8439-c7a7e5893394"). InnerVolumeSpecName "kube-api-access-c8srd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:59:14.036161 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.036130 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-model-cache\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:14.036161 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.036156 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-home\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:14.036161 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.036165 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8srd\" (UniqueName: \"kubernetes.io/projected/88c888db-8d47-4771-8439-c7a7e5893394-kube-api-access-c8srd\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:14.036405 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.036175 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-tmp-dir\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:14.036405 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.036185 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c888db-8d47-4771-8439-c7a7e5893394-dshm\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:14.036405 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.036194 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c888db-8d47-4771-8439-c7a7e5893394-tls-certs\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:14.595096 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.595059 2572 generic.go:356] "Generic (PLEG): container finished" podID="88c888db-8d47-4771-8439-c7a7e5893394" containerID="d7d29b85af91bb28e0614f72c8f48175a2f4dbbd5e630c766162605c80119ad5" exitCode=0 Jul 14 19:59:14.595278 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.595103 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" Jul 14 19:59:14.595278 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.595133 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" event={"ID":"88c888db-8d47-4771-8439-c7a7e5893394","Type":"ContainerDied","Data":"d7d29b85af91bb28e0614f72c8f48175a2f4dbbd5e630c766162605c80119ad5"} Jul 14 19:59:14.595278 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.595174 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb" event={"ID":"88c888db-8d47-4771-8439-c7a7e5893394","Type":"ContainerDied","Data":"4605d638e2acfa4f19579d5fbf9feb7668fee5ebee5c1b72368ab2e9061c2bb2"} Jul 14 19:59:14.595278 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.595192 2572 scope.go:117] "RemoveContainer" containerID="d7d29b85af91bb28e0614f72c8f48175a2f4dbbd5e630c766162605c80119ad5" Jul 14 19:59:14.604510 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.604492 2572 scope.go:117] "RemoveContainer" containerID="d7d29b85af91bb28e0614f72c8f48175a2f4dbbd5e630c766162605c80119ad5" Jul 14 19:59:14.604791 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:59:14.604773 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d29b85af91bb28e0614f72c8f48175a2f4dbbd5e630c766162605c80119ad5\": container with ID starting with d7d29b85af91bb28e0614f72c8f48175a2f4dbbd5e630c766162605c80119ad5 not found: ID does not exist" containerID="d7d29b85af91bb28e0614f72c8f48175a2f4dbbd5e630c766162605c80119ad5" Jul 14 19:59:14.604858 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.604804 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d29b85af91bb28e0614f72c8f48175a2f4dbbd5e630c766162605c80119ad5"} err="failed to get container status \"d7d29b85af91bb28e0614f72c8f48175a2f4dbbd5e630c766162605c80119ad5\": rpc error: code = NotFound desc = could not find container \"d7d29b85af91bb28e0614f72c8f48175a2f4dbbd5e630c766162605c80119ad5\": container with ID starting with d7d29b85af91bb28e0614f72c8f48175a2f4dbbd5e630c766162605c80119ad5 not found: ID does not exist" Jul 14 19:59:14.615111 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.615079 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb"] Jul 14 19:59:14.620476 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.620444 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-356b53/rate-test-model-356b53-kserve-6d85d66b9f-n6vtb"] Jul 14 19:59:14.717925 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.717905 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:14.737822 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.737800 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7431-140405581853728_4ed2b1ce-c446-40b1-95de-24c11fe8588a/test-curl-7431-140405581853728/0.log" Jul 14 19:59:14.761914 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.761893 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7431-140405581853728"] Jul 14 19:59:14.765539 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.765518 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7431-140405581853728"] Jul 14 19:59:14.843100 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.843066 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7jmv\" (UniqueName: \"kubernetes.io/projected/4ed2b1ce-c446-40b1-95de-24c11fe8588a-kube-api-access-s7jmv\") pod \"4ed2b1ce-c446-40b1-95de-24c11fe8588a\" (UID: \"4ed2b1ce-c446-40b1-95de-24c11fe8588a\") " Jul 14 19:59:14.844996 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.844975 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed2b1ce-c446-40b1-95de-24c11fe8588a-kube-api-access-s7jmv" (OuterVolumeSpecName: "kube-api-access-s7jmv") pod "4ed2b1ce-c446-40b1-95de-24c11fe8588a" (UID: "4ed2b1ce-c446-40b1-95de-24c11fe8588a"). InnerVolumeSpecName "kube-api-access-s7jmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:59:14.892118 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.892073 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7431-140405581853728"] Jul 14 19:59:14.892363 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.892351 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c888db-8d47-4771-8439-c7a7e5893394" containerName="main" Jul 14 19:59:14.892406 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.892364 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c888db-8d47-4771-8439-c7a7e5893394" containerName="main" Jul 14 19:59:14.892406 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.892375 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="628eae60-3d50-4d1e-966d-4d0a571d1ead" containerName="main" Jul 14 19:59:14.892406 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.892381 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="628eae60-3d50-4d1e-966d-4d0a571d1ead" containerName="main" Jul 14 19:59:14.892406 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.892400 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ed2b1ce-c446-40b1-95de-24c11fe8588a" containerName="test-curl-7431-140405581853728" Jul 14 19:59:14.892406 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.892407 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ed2b1ce-c446-40b1-95de-24c11fe8588a" containerName="test-curl-7431-140405581853728" Jul 14 19:59:14.892561 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.892464 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ed2b1ce-c446-40b1-95de-24c11fe8588a" containerName="test-curl-7431-140405581853728" Jul 14 19:59:14.892561 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.892476 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="628eae60-3d50-4d1e-966d-4d0a571d1ead" containerName="main" Jul 14 19:59:14.892561 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.892483 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c888db-8d47-4771-8439-c7a7e5893394" containerName="main" Jul 14 19:59:14.895247 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.895229 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:14.900781 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.900750 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140405581853728"] Jul 14 19:59:14.943923 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:14.943889 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7jmv\" (UniqueName: \"kubernetes.io/projected/4ed2b1ce-c446-40b1-95de-24c11fe8588a-kube-api-access-s7jmv\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:15.044413 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:15.044380 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjmj2\" (UniqueName: \"kubernetes.io/projected/2f0d8b93-7486-4041-9ef7-4ad71efc92b8-kube-api-access-zjmj2\") pod \"test-curl-7431-140405581853728\" (UID: \"2f0d8b93-7486-4041-9ef7-4ad71efc92b8\") " pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:15.145461 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:15.145382 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjmj2\" (UniqueName: \"kubernetes.io/projected/2f0d8b93-7486-4041-9ef7-4ad71efc92b8-kube-api-access-zjmj2\") pod \"test-curl-7431-140405581853728\" (UID: \"2f0d8b93-7486-4041-9ef7-4ad71efc92b8\") " pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:15.153318 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:15.153289 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjmj2\" (UniqueName: \"kubernetes.io/projected/2f0d8b93-7486-4041-9ef7-4ad71efc92b8-kube-api-access-zjmj2\") pod \"test-curl-7431-140405581853728\" (UID: \"2f0d8b93-7486-4041-9ef7-4ad71efc92b8\") " pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:15.193077 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:15.193026 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ed2b1ce-c446-40b1-95de-24c11fe8588a" path="/var/lib/kubelet/pods/4ed2b1ce-c446-40b1-95de-24c11fe8588a/volumes" Jul 14 19:59:15.193429 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:15.193416 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="628eae60-3d50-4d1e-966d-4d0a571d1ead" path="/var/lib/kubelet/pods/628eae60-3d50-4d1e-966d-4d0a571d1ead/volumes" Jul 14 19:59:15.193748 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:15.193737 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c888db-8d47-4771-8439-c7a7e5893394" path="/var/lib/kubelet/pods/88c888db-8d47-4771-8439-c7a7e5893394/volumes" Jul 14 19:59:15.204949 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:15.204925 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:15.329281 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:15.329254 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140405581853728"] Jul 14 19:59:15.331840 ip-10-0-138-70 kubenswrapper[2572]: W0714 19:59:15.331807 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f0d8b93_7486_4041_9ef7_4ad71efc92b8.slice/crio-8eddad361ccdc4f414902f9fb54093c6ec2ef0b6c0d8cccbf1128227505ab052 WatchSource:0}: Error finding container 8eddad361ccdc4f414902f9fb54093c6ec2ef0b6c0d8cccbf1128227505ab052: Status 404 returned error can't find the container with id 8eddad361ccdc4f414902f9fb54093c6ec2ef0b6c0d8cccbf1128227505ab052 Jul 14 19:59:15.602210 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:15.602175 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:15.602210 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:15.602191 2572 scope.go:117] "RemoveContainer" containerID="764746a9dbee63732fc72b2fde975b4d42d5701ba3a0dfdd8e33c6f06641cd07" Jul 14 19:59:15.603449 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:15.603422 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140405581853728" event={"ID":"2f0d8b93-7486-4041-9ef7-4ad71efc92b8","Type":"ContainerStarted","Data":"8eddad361ccdc4f414902f9fb54093c6ec2ef0b6c0d8cccbf1128227505ab052"} Jul 14 19:59:16.611223 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:16.611187 2572 generic.go:356] "Generic (PLEG): container finished" podID="2f0d8b93-7486-4041-9ef7-4ad71efc92b8" containerID="7aab0c171a1eeda9ef2cd788b2a8810e14091681a76eac8fc24a441c4018ceb9" exitCode=0 Jul 14 19:59:16.611624 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:16.611250 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140405581853728" event={"ID":"2f0d8b93-7486-4041-9ef7-4ad71efc92b8","Type":"ContainerDied","Data":"7aab0c171a1eeda9ef2cd788b2a8810e14091681a76eac8fc24a441c4018ceb9"} Jul 14 19:59:17.748587 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:17.748562 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:17.767021 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:17.767000 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7431-140405581853728_2f0d8b93-7486-4041-9ef7-4ad71efc92b8/test-curl-7431-140405581853728/0.log" Jul 14 19:59:17.787883 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:17.787856 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7431-140405581853728"] Jul 14 19:59:17.791407 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:17.791377 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7431-140405581853728"] Jul 14 19:59:17.868345 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:17.868260 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjmj2\" (UniqueName: \"kubernetes.io/projected/2f0d8b93-7486-4041-9ef7-4ad71efc92b8-kube-api-access-zjmj2\") pod \"2f0d8b93-7486-4041-9ef7-4ad71efc92b8\" (UID: \"2f0d8b93-7486-4041-9ef7-4ad71efc92b8\") " Jul 14 19:59:17.870277 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:17.870255 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0d8b93-7486-4041-9ef7-4ad71efc92b8-kube-api-access-zjmj2" (OuterVolumeSpecName: "kube-api-access-zjmj2") pod "2f0d8b93-7486-4041-9ef7-4ad71efc92b8" (UID: "2f0d8b93-7486-4041-9ef7-4ad71efc92b8"). InnerVolumeSpecName "kube-api-access-zjmj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:59:17.969568 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:17.969534 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjmj2\" (UniqueName: \"kubernetes.io/projected/2f0d8b93-7486-4041-9ef7-4ad71efc92b8-kube-api-access-zjmj2\") on node \"ip-10-0-138-70.ec2.internal\" DevicePath \"\"" Jul 14 19:59:18.481174 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:18.481143 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:18.581215 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:18.581189 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:18.620948 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:18.620921 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:18.621134 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:18.620920 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eddad361ccdc4f414902f9fb54093c6ec2ef0b6c0d8cccbf1128227505ab052" Jul 14 19:59:18.630557 ip-10-0-138-70 kubenswrapper[2572]: E0714 19:59:18.630530 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7431-140405581853728\" is forbidden: User \"system:node:ip-10-0-138-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-70.ec2.internal' and this object" podUID="2f0d8b93-7486-4041-9ef7-4ad71efc92b8" pod="opendatahub/test-curl-7431-140405581853728" Jul 14 19:59:19.188506 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:19.188473 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f0d8b93-7486-4041-9ef7-4ad71efc92b8" path="/var/lib/kubelet/pods/2f0d8b93-7486-4041-9ef7-4ad71efc92b8/volumes" Jul 14 19:59:24.251722 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:24.251680 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:24.353579 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:24.353550 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:28.583427 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:28.583393 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:28.683037 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:28.683006 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:28.826660 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:28.826629 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:31.132123 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:31.132094 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:59:33.090061 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:33.090009 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:39.255415 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:39.255369 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:39.356001 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:39.355971 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:43.806519 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:43.806486 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:44.053792 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:44.053761 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:44.787502 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:44.787466 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:44.887417 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:44.887376 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:53.523783 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:53.523751 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:54.708939 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:54.708909 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 19:59:58.224054 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:58.224013 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pflx5"] Jul 14 19:59:59.268316 ip-10-0-138-70 kubenswrapper[2572]: I0714 19:59:59.268276 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:06.088736 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:06.088702 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:06.162654 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:06.162625 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:10.535258 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:10.535226 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:10.602250 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:10.602219 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:10.691454 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:10.691421 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6479548764-4w7zb_2268ec04-5a0a-482c-9b77-35a7fcc01f0c/authorino/0.log" Jul 14 20:00:16.388613 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:16.388535 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9jbtj_4ff72c66-3b21-4ae2-927b-dea2c6531637/manager/0.log" Jul 14 20:00:16.518552 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:16.518522 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f9786d97d-c99zz_94ff3e11-5fd0-4d91-8268-ca0353d8bda8/copy-manifests/0.log" Jul 14 20:00:16.531301 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:16.531276 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f9786d97d-c99zz_94ff3e11-5fd0-4d91-8268-ca0353d8bda8/manager/0.log" Jul 14 20:00:20.511869 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:20.511833 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6479548764-4w7zb_2268ec04-5a0a-482c-9b77-35a7fcc01f0c/authorino/0.log" Jul 14 20:00:21.257358 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:21.257329 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-pflx5_4993e8f1-f4c4-43f2-910f-92cc3f74c2af/limitador/0.log" Jul 14 20:00:22.133493 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:22.133461 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d877944c-8glz9_4eb12cf4-fd60-4976-a21b-86e9d1f6628d/kube-auth-proxy/0.log" Jul 14 20:00:23.531519 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:23.531491 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-njn5p_34844f98-eada-430e-9065-27f1f550ea5a/main/0.log" Jul 14 20:00:23.657642 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:23.657610 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-9qn4b_55661263-8e16-43f8-9621-34185009df14/main/0.log" Jul 14 20:00:27.527244 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.527213 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bmp8z/must-gather-4m84r"] Jul 14 20:00:27.527636 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.527501 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f0d8b93-7486-4041-9ef7-4ad71efc92b8" containerName="test-curl-7431-140405581853728" Jul 14 20:00:27.527636 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.527512 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f0d8b93-7486-4041-9ef7-4ad71efc92b8" containerName="test-curl-7431-140405581853728" Jul 14 20:00:27.527636 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.527570 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f0d8b93-7486-4041-9ef7-4ad71efc92b8" containerName="test-curl-7431-140405581853728" Jul 14 20:00:27.530665 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.530644 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bmp8z/must-gather-4m84r" Jul 14 20:00:27.533239 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.533216 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bmp8z\"/\"kube-root-ca.crt\"" Jul 14 20:00:27.533483 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.533459 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bmp8z\"/\"default-dockercfg-skmfs\"" Jul 14 20:00:27.533610 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.533460 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bmp8z\"/\"openshift-service-ca.crt\"" Jul 14 20:00:27.538881 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.538859 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bmp8z/must-gather-4m84r"] Jul 14 20:00:27.543724 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.543690 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:27.545948 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.545925 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbvwg\" (UniqueName: \"kubernetes.io/projected/a48255e9-7fb9-4ca8-9225-ee45dc88e039-kube-api-access-sbvwg\") pod \"must-gather-4m84r\" (UID: \"a48255e9-7fb9-4ca8-9225-ee45dc88e039\") " pod="openshift-must-gather-bmp8z/must-gather-4m84r" Jul 14 20:00:27.546062 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.545975 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a48255e9-7fb9-4ca8-9225-ee45dc88e039-must-gather-output\") pod \"must-gather-4m84r\" (UID: \"a48255e9-7fb9-4ca8-9225-ee45dc88e039\") " pod="openshift-must-gather-bmp8z/must-gather-4m84r" Jul 14 20:00:27.642366 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.642332 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" limit=1024 Jul 14 20:00:27.647012 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.646994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a48255e9-7fb9-4ca8-9225-ee45dc88e039-must-gather-output\") pod \"must-gather-4m84r\" (UID: \"a48255e9-7fb9-4ca8-9225-ee45dc88e039\") " pod="openshift-must-gather-bmp8z/must-gather-4m84r" Jul 14 20:00:27.647136 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.647051 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbvwg\" (UniqueName: \"kubernetes.io/projected/a48255e9-7fb9-4ca8-9225-ee45dc88e039-kube-api-access-sbvwg\") pod \"must-gather-4m84r\" (UID: \"a48255e9-7fb9-4ca8-9225-ee45dc88e039\") " pod="openshift-must-gather-bmp8z/must-gather-4m84r" Jul 14 20:00:27.647322 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.647306 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a48255e9-7fb9-4ca8-9225-ee45dc88e039-must-gather-output\") pod \"must-gather-4m84r\" (UID: \"a48255e9-7fb9-4ca8-9225-ee45dc88e039\") " pod="openshift-must-gather-bmp8z/must-gather-4m84r" Jul 14 20:00:27.654615 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.654598 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbvwg\" (UniqueName: \"kubernetes.io/projected/a48255e9-7fb9-4ca8-9225-ee45dc88e039-kube-api-access-sbvwg\") pod \"must-gather-4m84r\" (UID: \"a48255e9-7fb9-4ca8-9225-ee45dc88e039\") " pod="openshift-must-gather-bmp8z/must-gather-4m84r" Jul 14 20:00:27.841145 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.841115 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bmp8z/must-gather-4m84r" Jul 14 20:00:27.970258 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:27.970178 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bmp8z/must-gather-4m84r"] Jul 14 20:00:27.972438 ip-10-0-138-70 kubenswrapper[2572]: W0714 20:00:27.972408 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda48255e9_7fb9_4ca8_9225_ee45dc88e039.slice/crio-ea1c7bbf5ec6ed8903b90d32a4dfa696f4d80b534a16c90d546b7ab05f691c96 WatchSource:0}: Error finding container ea1c7bbf5ec6ed8903b90d32a4dfa696f4d80b534a16c90d546b7ab05f691c96: Status 404 returned error can't find the container with id ea1c7bbf5ec6ed8903b90d32a4dfa696f4d80b534a16c90d546b7ab05f691c96 Jul 14 20:00:28.910150 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:28.910113 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bmp8z/must-gather-4m84r" event={"ID":"a48255e9-7fb9-4ca8-9225-ee45dc88e039","Type":"ContainerStarted","Data":"ea1c7bbf5ec6ed8903b90d32a4dfa696f4d80b534a16c90d546b7ab05f691c96"} Jul 14 20:00:29.919952 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:29.919908 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bmp8z/must-gather-4m84r" event={"ID":"a48255e9-7fb9-4ca8-9225-ee45dc88e039","Type":"ContainerStarted","Data":"c29ff86d26454f457899ba033351312597e2151b5da13cd7d2299ccd04987e41"} Jul 14 20:00:29.919952 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:29.919962 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bmp8z/must-gather-4m84r" event={"ID":"a48255e9-7fb9-4ca8-9225-ee45dc88e039","Type":"ContainerStarted","Data":"663fb82e145c4173f05dab5f3d22a6213f011fd30b5f32e5bfee6dd113d3b8ec"} Jul 14 20:00:29.938241 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:29.938173 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bmp8z/must-gather-4m84r" podStartSLOduration=1.593395214 podStartE2EDuration="2.93815125s" totalImagesPullingTime="1.344756036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:00:29.936373582 +0000 UTC m=+2979.301443944" watchObservedRunningTime="2026-07-14 20:00:29.93815125 +0000 UTC m=+2979.303221613" Jul 14 20:00:31.020025 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:31.019984 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t6nzh_eb3e121f-d03c-496e-8aad-591b078ec990/global-pull-secret-syncer/0.log" Jul 14 20:00:31.068888 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:31.068854 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-88fh2_466721f5-d454-479b-97b6-9a8aad590b61/konnectivity-agent/0.log" Jul 14 20:00:31.196875 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:31.196845 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-70.ec2.internal_b04179ee16bc51140974d528cab249f3/haproxy/0.log" Jul 14 20:00:36.070063 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:36.069431 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6479548764-4w7zb_2268ec04-5a0a-482c-9b77-35a7fcc01f0c/authorino/0.log" Jul 14 20:00:36.323773 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:36.323689 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-pflx5_4993e8f1-f4c4-43f2-910f-92cc3f74c2af/limitador/0.log" Jul 14 20:00:38.322103 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:38.322062 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jhvtj_de665ef3-1bee-4c65-b71c-7a35a9afe9e3/node-exporter/0.log" Jul 14 20:00:38.354114 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:38.354083 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jhvtj_de665ef3-1bee-4c65-b71c-7a35a9afe9e3/kube-rbac-proxy/0.log" Jul 14 20:00:38.378241 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:38.378134 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jhvtj_de665ef3-1bee-4c65-b71c-7a35a9afe9e3/init-textfile/0.log" Jul 14 20:00:38.893208 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:38.893165 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8576574855-l8cgz_e979de95-0f60-4ef1-868f-6ec4a12444f3/telemeter-client/0.log" Jul 14 20:00:38.918554 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:38.918519 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8576574855-l8cgz_e979de95-0f60-4ef1-868f-6ec4a12444f3/reload/0.log" Jul 14 20:00:38.940414 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:38.940386 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8576574855-l8cgz_e979de95-0f60-4ef1-868f-6ec4a12444f3/kube-rbac-proxy/0.log" Jul 14 20:00:43.203022 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:43.202989 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4kwvb_005d066e-4e27-42af-8e47-e6691b93f148/dns/0.log" Jul 14 20:00:43.222422 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:43.222385 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4kwvb_005d066e-4e27-42af-8e47-e6691b93f148/kube-rbac-proxy/0.log" Jul 14 20:00:43.329726 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:43.329636 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vq5gr_8b8debbe-5a4a-4bfc-bc3f-1ce3aa17f96f/dns-node-resolver/0.log" Jul 14 20:00:43.863748 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:43.863706 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gndd8_a6bb4d98-4bbc-4c23-82ad-351fdd921751/node-ca/0.log" Jul 14 20:00:44.405534 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.405505 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9"] Jul 14 20:00:44.411132 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.411106 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.418849 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.418822 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9"] Jul 14 20:00:44.503653 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.503609 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lwtb\" (UniqueName: \"kubernetes.io/projected/0d398479-806a-4c46-8260-906037a72bc8-kube-api-access-5lwtb\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.503944 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.503919 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d398479-806a-4c46-8260-906037a72bc8-podres\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.504109 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.504092 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d398479-806a-4c46-8260-906037a72bc8-sys\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.504297 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.504279 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d398479-806a-4c46-8260-906037a72bc8-proc\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.504436 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.504420 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d398479-806a-4c46-8260-906037a72bc8-lib-modules\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.605114 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.605082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lwtb\" (UniqueName: \"kubernetes.io/projected/0d398479-806a-4c46-8260-906037a72bc8-kube-api-access-5lwtb\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.605114 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.605128 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d398479-806a-4c46-8260-906037a72bc8-podres\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.605369 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.605150 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d398479-806a-4c46-8260-906037a72bc8-sys\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.605369 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.605195 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d398479-806a-4c46-8260-906037a72bc8-proc\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.605369 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.605229 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d398479-806a-4c46-8260-906037a72bc8-lib-modules\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.605369 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.605283 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d398479-806a-4c46-8260-906037a72bc8-sys\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.605369 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.605286 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d398479-806a-4c46-8260-906037a72bc8-podres\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.605369 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.605306 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d398479-806a-4c46-8260-906037a72bc8-proc\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.605369 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.605344 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d398479-806a-4c46-8260-906037a72bc8-lib-modules\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.614145 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.614118 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lwtb\" (UniqueName: \"kubernetes.io/projected/0d398479-806a-4c46-8260-906037a72bc8-kube-api-access-5lwtb\") pod \"perf-node-gather-daemonset-srkp9\" (UID: \"0d398479-806a-4c46-8260-906037a72bc8\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.723904 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.723810 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:44.902853 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:44.902819 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9"] Jul 14 20:00:44.908968 ip-10-0-138-70 kubenswrapper[2572]: W0714 20:00:44.908923 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0d398479_806a_4c46_8260_906037a72bc8.slice/crio-e049ca025759c1f8b272fae30e5522bb15d77b0588bb6d29bb32a57f89fea556 WatchSource:0}: Error finding container e049ca025759c1f8b272fae30e5522bb15d77b0588bb6d29bb32a57f89fea556: Status 404 returned error can't find the container with id e049ca025759c1f8b272fae30e5522bb15d77b0588bb6d29bb32a57f89fea556 Jul 14 20:00:45.019485 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:45.019328 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" event={"ID":"0d398479-806a-4c46-8260-906037a72bc8","Type":"ContainerStarted","Data":"e049ca025759c1f8b272fae30e5522bb15d77b0588bb6d29bb32a57f89fea556"} Jul 14 20:00:45.097784 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:45.097745 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d877944c-8glz9_4eb12cf4-fd60-4976-a21b-86e9d1f6628d/kube-auth-proxy/0.log" Jul 14 20:00:45.939527 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:45.939500 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-95h2r_b4b0f202-374a-4e71-97db-50a6422c7767/serve-healthcheck-canary/0.log" Jul 14 20:00:46.028027 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:46.027983 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" event={"ID":"0d398479-806a-4c46-8260-906037a72bc8","Type":"ContainerStarted","Data":"76394ecabcd1b48d7fc8d0e1ce2a3ef02e1674fae0d5f7f1b66a72ab130dbf6d"} Jul 14 20:00:46.028683 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:46.028654 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:46.044780 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:46.044712 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" podStartSLOduration=2.044691607 podStartE2EDuration="2.044691607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 20:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:00:46.043862531 +0000 UTC m=+2995.408932894" watchObservedRunningTime="2026-07-14 20:00:46.044691607 +0000 UTC m=+2995.409761970" Jul 14 20:00:46.458533 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:46.458505 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8zltv_645c204e-e9c0-4963-a53a-4192c14caaae/kube-rbac-proxy/0.log" Jul 14 20:00:46.477969 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:46.477942 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8zltv_645c204e-e9c0-4963-a53a-4192c14caaae/exporter/0.log" Jul 14 20:00:46.499511 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:46.499479 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8zltv_645c204e-e9c0-4963-a53a-4192c14caaae/extractor/0.log" Jul 14 20:00:49.049148 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:49.049091 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9jbtj_4ff72c66-3b21-4ae2-927b-dea2c6531637/manager/0.log" Jul 14 20:00:49.078615 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:49.078583 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f9786d97d-c99zz_94ff3e11-5fd0-4d91-8268-ca0353d8bda8/manager/0.log" Jul 14 20:00:49.097851 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:49.097821 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f9786d97d-c99zz_94ff3e11-5fd0-4d91-8268-ca0353d8bda8/copy-manifests/0.log" Jul 14 20:00:50.417581 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:50.417544 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-59cf65bbd6-ttggk_2cfa40f7-76f3-4f0b-ba97-3c2b96943e58/manager/0.log" Jul 14 20:00:51.256321 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:51.256293 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 20:00:51.258124 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:51.258089 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 20:00:51.488095 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:51.488034 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d877944c-8glz9_4eb12cf4-fd60-4976-a21b-86e9d1f6628d/kube-auth-proxy/0.log" Jul 14 20:00:52.046066 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:52.046006 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-srkp9" Jul 14 20:00:52.854016 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:52.853978 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d877944c-8glz9_4eb12cf4-fd60-4976-a21b-86e9d1f6628d/kube-auth-proxy/0.log" Jul 14 20:00:57.461927 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:57.461896 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbjqz_6cedffc7-b943-486f-9322-73a3b01c7d93/kube-multus-additional-cni-plugins/0.log" Jul 14 20:00:57.481506 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:57.481482 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbjqz_6cedffc7-b943-486f-9322-73a3b01c7d93/egress-router-binary-copy/0.log" Jul 14 20:00:57.499210 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:57.499180 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbjqz_6cedffc7-b943-486f-9322-73a3b01c7d93/cni-plugins/0.log" Jul 14 20:00:57.518711 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:57.518688 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbjqz_6cedffc7-b943-486f-9322-73a3b01c7d93/bond-cni-plugin/0.log" Jul 14 20:00:57.539998 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:57.539971 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbjqz_6cedffc7-b943-486f-9322-73a3b01c7d93/routeoverride-cni/0.log" Jul 14 20:00:57.559572 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:57.559547 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbjqz_6cedffc7-b943-486f-9322-73a3b01c7d93/whereabouts-cni-bincopy/0.log" Jul 14 20:00:57.582146 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:57.582118 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbjqz_6cedffc7-b943-486f-9322-73a3b01c7d93/whereabouts-cni/0.log" Jul 14 20:00:57.610706 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:57.610682 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dwftk_3dcd3c48-d0c0-4d6c-9681-af171f9a3567/kube-multus/0.log" Jul 14 20:00:57.693179 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:57.693152 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h7v99_6e91d78e-4060-4063-86ac-5d7be516f269/network-metrics-daemon/0.log" Jul 14 20:00:57.711653 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:57.711631 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h7v99_6e91d78e-4060-4063-86ac-5d7be516f269/kube-rbac-proxy/0.log" Jul 14 20:00:58.985107 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:58.985083 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-controller/0.log" Jul 14 20:00:59.002647 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:59.002621 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/0.log" Jul 14 20:00:59.002805 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:59.002768 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovn-acl-logging/1.log" Jul 14 20:00:59.022191 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:59.022167 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/kube-rbac-proxy-node/0.log" Jul 14 20:00:59.041952 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:59.041927 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/kube-rbac-proxy-ovn-metrics/0.log" Jul 14 20:00:59.058263 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:59.058242 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/northd/0.log" Jul 14 20:00:59.076384 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:59.076362 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/nbdb/0.log" Jul 14 20:00:59.094873 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:59.094848 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/sbdb/0.log" Jul 14 20:00:59.146480 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:00:59.146427 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xw4dp_b0286c27-b1d6-437f-90ff-5ef44af9d733/ovnkube-controller/0.log" Jul 14 20:01:00.226169 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:01:00.226139 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-h5x8l_dfb34251-8e7e-4321-9a87-0dd7af1a459c/network-check-target-container/0.log" Jul 14 20:01:01.352863 ip-10-0-138-70 kubenswrapper[2572]: I0714 20:01:01.352791 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-l65x7_0db1df7d-77b1-483f-92bb-8c9219e3b4e4/iptables-alerter/0.log"