Jul 09 18:25:32.838000 ip-10-0-138-253 systemd[1]: Starting Kubernetes Kubelet... Jul 09 18:25:33.250117 ip-10-0-138-253 systemd[1]: Started Kubernetes Kubelet. Jul 09 18:25:33.276776 ip-10-0-138-253 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 18:25:33.276776 ip-10-0-138-253 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 18:25:33.276776 ip-10-0-138-253 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 18:25:33.276776 ip-10-0-138-253 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 18:25:33.276776 ip-10-0-138-253 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 18:25:33.276776 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.185018 2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 18:25:33.276776 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189131 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:25:33.276776 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189142 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:25:33.276776 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189146 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:25:33.276776 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189149 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:25:33.276776 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189153 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:25:33.276776 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189156 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:25:33.276776 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189159 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:25:33.276776 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189162 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189165 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189168 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189171 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189173 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189176 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189179 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189181 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189193 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189197 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189200 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189202 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189205 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189208 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189210 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189213 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189216 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189218 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189221 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189224 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:25:33.277965 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189227 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189230 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189233 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189235 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189238 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189241 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189243 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189246 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189251 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189255 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189259 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189262 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189265 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189268 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189270 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189273 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189276 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189278 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189281 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:25:33.279205 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189284 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189287 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189290 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189292 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189295 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189298 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189300 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189302 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189305 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189307 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189311 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189314 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189317 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189320 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189323 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189326 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189328 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189331 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189334 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:25:33.280271 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189336 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189339 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189341 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189344 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189346 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189349 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189351 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189354 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189356 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189359 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189364 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189367 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189370 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189372 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189375 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189378 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189382 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189386 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189389 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189392 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:25:33.281298 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189395 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189398 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189400 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189403 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189406 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189409 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189412 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189415 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189418 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189421 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189423 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189426 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189429 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.189432 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190160 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190168 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190172 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190175 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190177 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190180 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:25:33.282261 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190183 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190185 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190188 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190193 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190197 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190199 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190202 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190205 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190207 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190210 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190212 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190215 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190217 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190220 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190223 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190225 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190228 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190230 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190233 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:25:33.283217 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190236 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190239 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190241 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190244 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190246 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190249 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190251 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190256 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190259 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190262 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190264 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190267 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190271 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190274 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190277 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190281 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190284 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190287 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190290 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190293 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:25:33.284151 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190296 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190299 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190301 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190303 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190306 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190308 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190311 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190313 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190316 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190318 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190321 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190324 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190326 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190329 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190331 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190334 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190336 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190339 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190342 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190347 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:25:33.284820 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190349 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190354 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190356 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190359 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190362 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190364 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190368 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190370 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190373 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190375 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190378 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190381 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190383 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190386 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190388 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190391 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190393 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190397 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190400 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190404 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:25:33.285338 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190407 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190409 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190412 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190415 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190418 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190421 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190423 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190427 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190429 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190432 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190435 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190439 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190441 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190445 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190838 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190843 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190846 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190849 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190852 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:25:33.285929 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190855 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190857 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190860 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190863 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190866 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190868 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190871 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190874 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190876 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190879 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190881 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190884 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190886 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190889 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190891 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190894 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190897 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190899 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190901 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190904 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:25:33.286980 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190906 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190909 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190912 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190914 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190918 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190921 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190925 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190927 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190930 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190932 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190935 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190939 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190943 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190945 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190948 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190951 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190955 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190957 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190960 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190963 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:25:33.288060 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190965 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190967 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190970 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190973 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190977 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190980 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190984 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190987 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190990 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190994 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190996 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.190999 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191001 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191004 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191006 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191009 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191012 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191015 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191020 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:25:33.289131 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191023 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191026 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191028 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191031 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191033 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191036 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191038 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191041 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191043 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191046 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191049 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191051 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191054 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191057 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191059 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191062 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191064 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191067 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191069 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:25:33.289951 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191072 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191074 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191077 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191079 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191083 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191085 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191088 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191090 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191092 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191095 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191099 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191101 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191105 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191108 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191111 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191113 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191175 2558 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191181 2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191188 2558 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191192 2558 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191195 2558 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 18:25:33.291002 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191199 2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191202 2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191206 2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191211 2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191214 2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191217 2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191220 2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191223 2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191226 2558 flags.go:64] FLAG: --cgroup-root="" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191229 2558 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191232 2558 flags.go:64] FLAG: --client-ca-file="" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191235 2558 flags.go:64] FLAG: --cloud-provider="external" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191237 2558 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191241 2558 flags.go:64] FLAG: --cluster-domain="" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191244 2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191247 2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191250 2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191253 2558 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191257 2558 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191259 2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191262 2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191267 2558 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191270 2558 flags.go:64] FLAG: --contention-profiling="false" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191274 2558 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 18:25:33.292259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191277 2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191280 2558 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191283 2558 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191287 2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191289 2558 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191292 2558 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191295 2558 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191298 2558 flags.go:64] FLAG: --enable-server="true" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191301 2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191305 2558 flags.go:64] FLAG: --event-burst="100" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191308 2558 flags.go:64] FLAG: --event-qps="50" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191311 2558 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191314 2558 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191317 2558 flags.go:64] FLAG: --eviction-hard="" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191321 2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191324 2558 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191327 2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191330 2558 flags.go:64] FLAG: --eviction-soft="" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191333 2558 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191335 2558 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191338 2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191341 2558 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191343 2558 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191346 2558 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191349 2558 flags.go:64] FLAG: --feature-gates="" Jul 09 18:25:33.293307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191353 2558 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191355 2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191358 2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191361 2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191366 2558 flags.go:64] FLAG: --healthz-port="10248" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191369 2558 flags.go:64] FLAG: --help="false" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191374 2558 flags.go:64] FLAG: --hostname-override="ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191377 2558 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191380 2558 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191383 2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191386 2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191390 2558 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191393 2558 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191395 2558 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191399 2558 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191402 2558 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191405 2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191408 2558 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191410 2558 flags.go:64] FLAG: --kube-reserved="" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191413 2558 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191416 2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191420 2558 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191422 2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191425 2558 flags.go:64] FLAG: --lock-file="" Jul 09 18:25:33.294162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191428 2558 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191431 2558 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191434 2558 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191439 2558 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191441 2558 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191444 2558 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191446 2558 flags.go:64] FLAG: --logging-format="text" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191449 2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191452 2558 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191455 2558 flags.go:64] FLAG: --manifest-url="" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191458 2558 flags.go:64] FLAG: --manifest-url-header="" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191462 2558 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191468 2558 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191472 2558 flags.go:64] FLAG: --max-pods="110" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191476 2558 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191479 2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191482 2558 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191485 2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191487 2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191490 2558 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191493 2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191500 2558 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191503 2558 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191506 2558 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 18:25:33.414742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191509 2558 flags.go:64] FLAG: --pod-cidr="" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191511 2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191518 2558 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191520 2558 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191523 2558 flags.go:64] FLAG: --pods-per-core="0" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191528 2558 flags.go:64] FLAG: --port="10250" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191531 2558 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191534 2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06ce12f199fdb33bf" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191536 2558 flags.go:64] FLAG: --qos-reserved="" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191539 2558 flags.go:64] FLAG: --read-only-port="10255" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191542 2558 flags.go:64] FLAG: --register-node="true" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191545 2558 flags.go:64] FLAG: --register-with-taints="" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191549 2558 flags.go:64] FLAG: --registry-burst="10" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191551 2558 flags.go:64] FLAG: --registry-qps="5" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191554 2558 flags.go:64] FLAG: --reserved-cpus="" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191557 2558 flags.go:64] FLAG: --reserved-memory="" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191561 2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191564 2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191567 2558 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191570 2558 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191572 2558 flags.go:64] FLAG: --runonce="false" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191576 2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191581 2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191584 2558 flags.go:64] FLAG: --seccomp-default="false" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191587 2558 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 18:25:33.456381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191589 2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191592 2558 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191595 2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191598 2558 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191601 2558 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191604 2558 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191606 2558 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191609 2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191612 2558 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191615 2558 flags.go:64] FLAG: --system-cgroups="" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191618 2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191623 2558 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191626 2558 flags.go:64] FLAG: --tls-cert-file="" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191629 2558 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191633 2558 flags.go:64] FLAG: --tls-min-version="" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191636 2558 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191639 2558 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191641 2558 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191644 2558 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191647 2558 flags.go:64] FLAG: --v="2" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191654 2558 flags.go:64] FLAG: --version="false" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191658 2558 flags.go:64] FLAG: --vmodule="" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191661 2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.191664 2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191783 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:25:33.463765 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191787 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191790 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191793 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191797 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191802 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191805 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191807 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191810 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191813 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191815 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191818 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191821 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191823 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191825 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191828 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191831 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191834 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191836 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191839 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191841 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:25:33.464681 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191844 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191847 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191850 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191852 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191855 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191857 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191859 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191862 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191864 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191867 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191869 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191872 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191874 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191877 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191879 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191883 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191888 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191890 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191893 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191895 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:25:33.465592 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191898 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191901 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191905 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191908 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191911 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191914 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191918 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191921 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191924 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191926 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191929 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191932 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191934 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191937 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191941 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191944 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191948 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191951 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191953 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:25:33.466460 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191956 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191959 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191961 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191964 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191967 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191970 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191972 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191975 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191979 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191983 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191985 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191988 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191991 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191993 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191996 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.191999 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192001 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192004 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192006 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192009 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:25:33.467178 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192012 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192015 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192018 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192021 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192024 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192026 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192029 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192032 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192035 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192038 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192040 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192044 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192047 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192049 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192052 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192054 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192057 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192059 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:25:33.467818 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.192062 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:25:33.468348 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.192673 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 18:25:33.468348 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.199295 2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 09 18:25:33.468348 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.199310 2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 18:25:33.468348 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.199329 2558 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 18:25:33.468348 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.199335 2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 18:25:33.468348 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199388 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:25:33.468348 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199396 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:25:33.468348 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199400 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:25:33.468348 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199403 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:25:33.468348 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199406 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:25:33.468348 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199408 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:25:33.468348 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199411 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:25:33.468348 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199414 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:25:33.468348 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199416 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199419 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199422 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199424 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199427 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199429 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199432 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199434 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199437 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199439 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199442 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199444 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199447 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199449 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199452 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199454 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199457 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199460 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199462 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199465 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:25:33.468869 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199467 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199471 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199474 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199476 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199479 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199481 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199484 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199487 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199490 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199493 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199495 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199498 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199500 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199503 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199505 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199508 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199510 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199513 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199515 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199518 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:25:33.469444 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199520 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199523 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199526 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199529 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199531 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199533 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199536 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199538 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199541 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199543 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199546 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199548 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199552 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199556 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199560 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199563 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199565 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199568 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199570 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:25:33.470087 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199573 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199575 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199578 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199580 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199583 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199585 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199588 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199590 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199593 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199595 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199598 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199600 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199603 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199605 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199608 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199611 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199613 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199617 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199621 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199623 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:25:33.470663 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199626 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199629 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199632 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199634 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199637 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199642 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199645 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199648 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199651 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199653 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199656 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199659 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.199664 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199799 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199805 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:25:33.471295 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199809 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199813 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199816 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199818 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199821 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199824 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199826 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199829 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199831 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199834 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199836 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199839 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199841 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199844 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199846 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199849 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199853 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199855 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199858 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199861 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199863 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:25:33.472125 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199866 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199869 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199873 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199875 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199878 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199881 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199884 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199886 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199889 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199892 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199894 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199897 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199899 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199902 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199905 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199909 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199912 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199915 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199918 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199920 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:25:33.473077 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199923 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199925 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199928 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199930 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199933 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199935 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199938 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199945 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199947 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199950 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199952 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199955 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199957 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199960 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199964 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199966 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199969 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199972 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199974 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199977 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:25:33.473788 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199980 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199982 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199985 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199987 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199990 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199992 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199995 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.199998 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200000 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200003 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200005 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200007 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200010 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200013 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200015 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200017 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200020 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200022 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200025 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:25:33.474469 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200029 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200032 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200035 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200038 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200041 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200043 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200046 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200050 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200052 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200055 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200058 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200060 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200063 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200066 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200068 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200071 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:25:33.475145 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:33.200073 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:25:33.476077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.200078 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 18:25:33.476077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.200172 2558 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 18:25:33.476077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.202722 2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 18:25:33.476077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.203962 2558 server.go:1019] "Starting client certificate rotation" Jul 09 18:25:33.476077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.204058 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 18:25:33.476077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.204112 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 18:25:33.476077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.206278 2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 18:25:33.476077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.208428 2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 18:25:33.476077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.214417 2558 log.go:25] "Validated CRI v1 runtime API" Jul 09 18:25:33.476077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.218353 2558 log.go:25] "Validated CRI v1 image API" Jul 09 18:25:33.476077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.218978 2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 18:25:33.476077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.220958 2558 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b2ee1001-9c5d-410d-9425-8dcb2ad39b6f:/dev/nvme0n1p4 dee9e364-ecbd-447b-98f5-1b73ec214d42:/dev/nvme0n1p3] Jul 09 18:25:33.476538 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.220976 2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 18:25:33.476762 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.226120 2558 manager.go:216] Machine: {Timestamp:2026-07-09 18:25:33.225027102 +0000 UTC m=+0.289692767 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096915 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fc798389a17fcb00e0d33ebb86eb6 SystemUUID:ec2fc798-389a-17fc-b00e-0d33ebb86eb6 BootID:a1a9284f-bd4c-4371-b9ec-154c2d62aa42 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3b:d7:48:33:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3b:d7:48:33:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:15:4e:bb:3c:30 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 18:25:33.476762 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.226220 2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 18:25:33.476762 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.226292 2558 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 18:25:33.476762 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.229193 2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 18:25:33.476762 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.229215 2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-253.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 18:25:33.476762 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.230500 2558 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 18:25:33.476762 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.230510 2558 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 18:25:33.476762 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.230518 2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.231192 2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.232544 2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.236853 2558 state_mem.go:36] "Initialized new in-memory state store" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.236960 2558 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.237683 2558 kubelet.go:486] "Attempting to sync node with API server" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.237693 2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.237719 2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.237727 2558 kubelet.go:398] "Adding apiserver pod source" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.237737 2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.238026 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.238040 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.239920 2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.241945 2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.241973 2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.242126 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.242144 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.242151 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.242160 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.242169 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.242176 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.242669 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.242677 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.242684 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.242689 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.242716 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.242724 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.244200 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.244207 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.244894 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.249210 2558 server.go:1268] "Started kubelet" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.249281 2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.249975 2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.250029 2558 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.254096 2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.254717 2558 server.go:310] "Adding debug handlers to kubelet server" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.254866 2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-253.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.254723 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.255249 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 18:25:33.604153 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.256624 2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.260252 2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.260263 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.260347 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.260858 2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.260868 2558 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.260885 2558 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.260988 2558 reconstruct.go:97] "Volume reconstruction finished" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.260999 2558 reconciler.go:29] "Reconciler: start to sync state" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.261029 2558 factory.go:55] Registering systemd factory Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.261046 2558 factory.go:223] Registration of the systemd container factory successfully Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.261275 2558 factory.go:153] Registering CRI-O factory Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.261285 2558 factory.go:223] Registration of the crio container factory successfully Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.261324 2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.261344 2558 factory.go:103] Registering Raw factory Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.261356 2558 manager.go:1196] Started watching for new ooms in manager Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.261735 2558 manager.go:349] Starting recovery of all containers Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.272835 2558 manager.go:354] Recovery completed Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.276975 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.277862 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.276911 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-253.ec2.internal.18c0b2559f0e1da2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-253.ec2.internal,UID:ip-10-0-138-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-253.ec2.internal,},FirstTimestamp:2026-07-09 18:25:33.249183138 +0000 UTC m=+0.313848803,LastTimestamp:2026-07-09 18:25:33.249183138 +0000 UTC m=+0.313848803,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-253.ec2.internal,}" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.278209 2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.279469 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.279504 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.279516 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.280148 2558 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.280158 2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.280169 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.280172 2558 state_mem.go:36] "Initialized new in-memory state store" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.280907 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.280924 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.280937 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.282479 2558 policy_none.go:49] "None policy: Start" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.282492 2558 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 18:25:33.605479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.282501 2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 18:25:33.608000 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.284546 2558 policy_none.go:47] "Start" Jul 09 18:25:33.608000 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.289407 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-253.ec2.internal.18c0b255a0dc6df3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-253.ec2.internal,UID:ip-10-0-138-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-253.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-253.ec2.internal,},FirstTimestamp:2026-07-09 18:25:33.279481331 +0000 UTC m=+0.344146999,LastTimestamp:2026-07-09 18:25:33.279481331 +0000 UTC m=+0.344146999,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-253.ec2.internal,}" Jul 09 18:25:33.608000 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.294288 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-253.ec2.internal.18c0b255a0dcdd32 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-253.ec2.internal,UID:ip-10-0-138-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-253.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-253.ec2.internal,},FirstTimestamp:2026-07-09 18:25:33.27950981 +0000 UTC m=+0.344175478,LastTimestamp:2026-07-09 18:25:33.27950981 +0000 UTC m=+0.344175478,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-253.ec2.internal,}" Jul 09 18:25:33.608000 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.301046 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-253.ec2.internal.18c0b255a0dd08de default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-253.ec2.internal,UID:ip-10-0-138-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-253.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-253.ec2.internal,},FirstTimestamp:2026-07-09 18:25:33.27952099 +0000 UTC m=+0.344186657,LastTimestamp:2026-07-09 18:25:33.27952099 +0000 UTC m=+0.344186657,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-253.ec2.internal,}" Jul 09 18:25:33.608000 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.301145 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nplmw" Jul 09 18:25:33.608000 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.311408 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nplmw" Jul 09 18:25:33.608000 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.317077 2558 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-253.ec2.internal.18c0b255a0dc6df3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-253.ec2.internal.18c0b255a0dc6df3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-253.ec2.internal,UID:ip-10-0-138-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-253.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-253.ec2.internal,},FirstTimestamp:2026-07-09 18:25:33.279481331 +0000 UTC m=+0.344146999,LastTimestamp:2026-07-09 18:25:33.280917337 +0000 UTC m=+0.345583003,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-253.ec2.internal,}" Jul 09 18:25:33.608000 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.322619 2558 manager.go:337] "Starting Device Plugin manager" Jul 09 18:25:33.608000 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.322645 2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 18:25:33.608000 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.322654 2558 server.go:87] "Starting device plugin registration server" Jul 09 18:25:33.608000 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.322876 2558 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 18:25:33.608000 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.322886 2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.322999 2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.323091 2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.323100 2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.323519 2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.323558 2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.424398 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.425593 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.425625 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.425636 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.425666 2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.434787 2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.434806 2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-253.ec2.internal\": node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.483500 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.551592 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.552946 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.552990 2558 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.553016 2558 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.553053 2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.561912 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:25:33.608532 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.584576 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.654763 2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-253.ec2.internal"] Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.654844 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.656655 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.656693 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.656705 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.657892 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.658060 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.658091 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.658690 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.658736 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.658748 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.659165 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.659191 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.659201 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.660215 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.660238 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.660983 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.661025 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.661039 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.662115 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c6925309b7197f6ab1e3cdd371df9a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal\" (UID: \"2c6925309b7197f6ab1e3cdd371df9a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.662137 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c5edb6b0989376dda6e5d5712ab11e9-config\") pod \"kube-apiserver-proxy-ip-10-0-138-253.ec2.internal\" (UID: \"8c5edb6b0989376dda6e5d5712ab11e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.662182 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c6925309b7197f6ab1e3cdd371df9a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal\" (UID: \"2c6925309b7197f6ab1e3cdd371df9a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.685152 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.694610 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-253.ec2.internal\" not found" node="ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.764166 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.699865 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-253.ec2.internal\" not found" node="ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.765398 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.762822 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c5edb6b0989376dda6e5d5712ab11e9-config\") pod \"kube-apiserver-proxy-ip-10-0-138-253.ec2.internal\" (UID: \"8c5edb6b0989376dda6e5d5712ab11e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.765398 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.762862 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c6925309b7197f6ab1e3cdd371df9a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal\" (UID: \"2c6925309b7197f6ab1e3cdd371df9a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.765398 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.762889 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c6925309b7197f6ab1e3cdd371df9a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal\" (UID: \"2c6925309b7197f6ab1e3cdd371df9a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.765398 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.762928 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c5edb6b0989376dda6e5d5712ab11e9-config\") pod \"kube-apiserver-proxy-ip-10-0-138-253.ec2.internal\" (UID: \"8c5edb6b0989376dda6e5d5712ab11e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.765398 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.762939 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c6925309b7197f6ab1e3cdd371df9a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal\" (UID: \"2c6925309b7197f6ab1e3cdd371df9a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.765398 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.762945 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c6925309b7197f6ab1e3cdd371df9a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal\" (UID: \"2c6925309b7197f6ab1e3cdd371df9a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal" Jul 09 18:25:33.911631 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.786208 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:33.911631 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.886341 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:34.036388 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:33.987846 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:34.036388 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:33.996627 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal" Jul 09 18:25:34.036388 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.002793 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-253.ec2.internal" Jul 09 18:25:34.121620 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:34.089528 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:34.126099 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.124721 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:25:34.218515 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:34.191398 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:34.218515 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.203404 2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 18:25:34.218515 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.203537 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 18:25:34.218515 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.203565 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 18:25:34.389632 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:34.292271 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:34.389632 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.295609 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:25:34.389632 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.296561 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:25:34.389632 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.296592 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:25:34.389632 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.296602 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:25:34.389632 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.315801 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 18:20:33 +0000 UTC" deadline="2028-02-07 21:40:48.23113691 +0000 UTC" Jul 09 18:25:34.389632 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.315824 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13875h15m13.915315839s" Jul 09 18:25:34.393475 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:34.393321 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:34.532217 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:34.493377 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:34.645314 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:34.594113 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:34.738423 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:34.693876 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5edb6b0989376dda6e5d5712ab11e9.slice/crio-ec038d7c1b551cc7420a15d3a974ddd1f293a44f0f6cc6badbce2a19c8a60fd1 WatchSource:0}: Error finding container ec038d7c1b551cc7420a15d3a974ddd1f293a44f0f6cc6badbce2a19c8a60fd1: Status 404 returned error can't find the container with id ec038d7c1b551cc7420a15d3a974ddd1f293a44f0f6cc6badbce2a19c8a60fd1 Jul 09 18:25:34.738423 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:34.694175 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-253.ec2.internal\" not found" Jul 09 18:25:34.738423 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:34.694422 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6925309b7197f6ab1e3cdd371df9a5.slice/crio-dbf81ba0b4327abc9c96371e03a0333586a95b1f17ee56f38dbcf0b5795fe1fc WatchSource:0}: Error finding container dbf81ba0b4327abc9c96371e03a0333586a95b1f17ee56f38dbcf0b5795fe1fc: Status 404 returned error can't find the container with id dbf81ba0b4327abc9c96371e03a0333586a95b1f17ee56f38dbcf0b5795fe1fc Jul 09 18:25:34.738423 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.699629 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 18:25:34.868124 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.752427 2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:25:34.868124 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.761337 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal" Jul 09 18:25:34.868124 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.779302 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 18:25:34.868124 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.782405 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-253.ec2.internal" Jul 09 18:25:34.868124 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.799606 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 18:25:34.868124 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:34.855071 2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:25:35.239610 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.239521 2558 apiserver.go:52] "Watching apiserver" Jul 09 18:25:35.250999 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.250968 2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 18:25:35.252398 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.252029 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-lfpjv","kube-system/global-pull-secret-syncer-r7g5z","kube-system/konnectivity-agent-9wfz4","kube-system/kube-apiserver-proxy-ip-10-0-138-253.ec2.internal","openshift-dns/node-resolver-tvjd5","openshift-multus/multus-additional-cni-plugins-qtt72","openshift-multus/multus-dzbx9","openshift-ovn-kubernetes/ovnkube-node-bmxck","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88","openshift-cluster-node-tuning-operator/tuned-vnnp7","openshift-image-registry/node-ca-4b29x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal","openshift-multus/network-metrics-daemon-r5jrm","openshift-network-diagnostics/network-check-target-j2xsk"] Jul 09 18:25:35.254671 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.254649 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.257196 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.257037 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:35.257196 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.257149 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:25:35.259564 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.259360 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:25:35.259666 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.259577 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9wfz4" Jul 09 18:25:35.259666 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.259641 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-585g8\"" Jul 09 18:25:35.259801 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.259695 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 18:25:35.261471 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.261452 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 18:25:35.262077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.261785 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:35.262077 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.261850 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:25:35.264749 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.264628 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tvjd5" Jul 09 18:25:35.267053 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.267033 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.267572 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.267550 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 18:25:35.267572 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.267570 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 18:25:35.267781 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.267695 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2jvbp\"" Jul 09 18:25:35.269498 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.269266 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.272217 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.271914 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.273367 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273347 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-kubernetes\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.273462 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273381 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-run\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.273462 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273407 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-var-lib-kubelet\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.273462 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273432 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f352f25-0ced-43ed-bac7-7579314f9f89-agent-certs\") pod \"konnectivity-agent-9wfz4\" (UID: \"5f352f25-0ced-43ed-bac7-7579314f9f89\") " pod="kube-system/konnectivity-agent-9wfz4" Jul 09 18:25:35.273629 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273482 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6fxn\" (UniqueName: \"kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn\") pod \"network-check-target-j2xsk\" (UID: \"0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e\") " pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:35.273629 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273513 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-modprobe-d\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.273629 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273539 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-systemd\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.273629 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273560 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-sys\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.273629 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273583 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-lib-modules\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.273629 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273607 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs\") pod \"network-metrics-daemon-r5jrm\" (UID: \"e673ec85-406b-4221-8e0a-441d1886d20d\") " pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:35.273922 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273634 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-sysconfig\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.273922 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273681 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-sysctl-conf\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.273922 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273706 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-host\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.273922 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273730 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f59262e-fc1c-4f6d-976d-228bc6cfe761-tmp\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.273922 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273756 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sprq9\" (UniqueName: \"kubernetes.io/projected/e673ec85-406b-4221-8e0a-441d1886d20d-kube-api-access-sprq9\") pod \"network-metrics-daemon-r5jrm\" (UID: \"e673ec85-406b-4221-8e0a-441d1886d20d\") " pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:35.273922 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273856 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-sysctl-d\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.273922 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273884 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-tuned\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.273922 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273913 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv8l4\" (UniqueName: \"kubernetes.io/projected/8f59262e-fc1c-4f6d-976d-228bc6cfe761-kube-api-access-wv8l4\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.274324 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.273938 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f352f25-0ced-43ed-bac7-7579314f9f89-konnectivity-ca\") pod \"konnectivity-agent-9wfz4\" (UID: \"5f352f25-0ced-43ed-bac7-7579314f9f89\") " pod="kube-system/konnectivity-agent-9wfz4" Jul 09 18:25:35.275185 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.275164 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.277659 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.277636 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lfpjv" Jul 09 18:25:35.280951 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.280932 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4b29x" Jul 09 18:25:35.282401 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.281926 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 18:25:35.282401 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.281970 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 18:25:35.282401 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.282237 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bdzsk\"" Jul 09 18:25:35.283746 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.283503 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:35.283746 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.283574 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:25:35.283746 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.283695 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 18:25:35.283746 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.283731 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 18:25:35.285446 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.285425 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 18:25:35.285542 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.285432 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-258kw\"" Jul 09 18:25:35.285607 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.285580 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 18:25:35.286063 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.286043 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 18:25:35.286158 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.286114 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 18:25:35.286158 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.286151 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 18:25:35.287155 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.287132 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 18:25:35.287245 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.287210 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 18:25:35.287427 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.287391 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 18:25:35.287572 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.287548 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m4wfl\"" Jul 09 18:25:35.287682 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.287662 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 18:25:35.287815 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.287744 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 18:25:35.287913 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.287898 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 18:25:35.287968 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.287930 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:25:35.288096 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.288060 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 18:25:35.288336 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.288316 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 18:25:35.288431 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.288318 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h2mrl\"" Jul 09 18:25:35.288549 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.288523 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5mfs6\"" Jul 09 18:25:35.289051 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.288661 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 18:25:35.289051 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.288732 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7zgpl\"" Jul 09 18:25:35.289051 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.288934 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 18:25:35.289051 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.288973 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r4cnv\"" Jul 09 18:25:35.289051 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.288984 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 18:25:35.289367 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.289155 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 18:25:35.289733 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.289696 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 18:25:35.304524 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.304497 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 18:25:35.353955 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.353929 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4d4gz" Jul 09 18:25:35.361923 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.361859 2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 18:25:35.369275 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.369251 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:25:35.369636 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.369615 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4d4gz" Jul 09 18:25:35.374536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374120 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-run-k8s-cni-cncf-io\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.374536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374161 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-kubelet\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.374536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374189 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8k9q\" (UniqueName: \"kubernetes.io/projected/9256aa17-feeb-4e97-a4cc-af4488848183-kube-api-access-k8k9q\") pod \"iptables-alerter-lfpjv\" (UID: \"9256aa17-feeb-4e97-a4cc-af4488848183\") " pod="openshift-network-operator/iptables-alerter-lfpjv" Jul 09 18:25:35.374536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374215 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-multus-cni-dir\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.374536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374245 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-sys\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.374536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374272 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6eea972-8391-4ba7-b1ce-9de56782baf3-cni-binary-copy\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.374536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374343 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-sys\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.374536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374353 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-env-overrides\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.374536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374385 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-dbus\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:35.374536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374418 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a93812e1-6915-47c8-b58a-d28334ab03a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.374536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374456 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x65k4\" (UniqueName: \"kubernetes.io/projected/a93812e1-6915-47c8-b58a-d28334ab03a3-kube-api-access-x65k4\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.374536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374497 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47cb0898-a067-4eb1-afe9-e2b5413b2cee-tmp-dir\") pod \"node-resolver-tvjd5\" (UID: \"47cb0898-a067-4eb1-afe9-e2b5413b2cee\") " pod="openshift-dns/node-resolver-tvjd5" Jul 09 18:25:35.374536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374538 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-multus-conf-dir\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.375267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374569 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-etc-kubernetes\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.375267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374594 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-run-ovn\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.375267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374625 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-log-socket\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.375267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374650 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.375267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374677 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9256aa17-feeb-4e97-a4cc-af4488848183-iptables-alerter-script\") pod \"iptables-alerter-lfpjv\" (UID: \"9256aa17-feeb-4e97-a4cc-af4488848183\") " pod="openshift-network-operator/iptables-alerter-lfpjv" Jul 09 18:25:35.375267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374708 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-sysctl-d\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.375267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374767 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-tuned\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.375267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374792 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f352f25-0ced-43ed-bac7-7579314f9f89-konnectivity-ca\") pod \"konnectivity-agent-9wfz4\" (UID: \"5f352f25-0ced-43ed-bac7-7579314f9f89\") " pod="kube-system/konnectivity-agent-9wfz4" Jul 09 18:25:35.375267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374819 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47cb0898-a067-4eb1-afe9-e2b5413b2cee-hosts-file\") pod \"node-resolver-tvjd5\" (UID: \"47cb0898-a067-4eb1-afe9-e2b5413b2cee\") " pod="openshift-dns/node-resolver-tvjd5" Jul 09 18:25:35.375267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374850 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-var-lib-kubelet\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.375267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374876 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-node-log\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.375267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374901 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a93812e1-6915-47c8-b58a-d28334ab03a3-os-release\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.375267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374932 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fxn\" (UniqueName: \"kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn\") pod \"network-check-target-j2xsk\" (UID: \"0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e\") " pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:35.375267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.374958 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-ovnkube-config\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.375972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375078 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/03116b04-5653-4707-8b77-a6c937981f49-serviceca\") pod \"node-ca-4b29x\" (UID: \"03116b04-5653-4707-8b77-a6c937981f49\") " pod="openshift-image-registry/node-ca-4b29x" Jul 09 18:25:35.375972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375107 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9256aa17-feeb-4e97-a4cc-af4488848183-host-slash\") pod \"iptables-alerter-lfpjv\" (UID: \"9256aa17-feeb-4e97-a4cc-af4488848183\") " pod="openshift-network-operator/iptables-alerter-lfpjv" Jul 09 18:25:35.375972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375132 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-cnibin\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.375972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375161 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs\") pod \"network-metrics-daemon-r5jrm\" (UID: \"e673ec85-406b-4221-8e0a-441d1886d20d\") " pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:35.375972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375192 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-hostroot\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.375972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375217 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-run-multus-certs\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.375972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375242 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a93812e1-6915-47c8-b58a-d28334ab03a3-cnibin\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.375972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375269 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f59262e-fc1c-4f6d-976d-228bc6cfe761-tmp\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.375972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375298 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprq9\" (UniqueName: \"kubernetes.io/projected/e673ec85-406b-4221-8e0a-441d1886d20d-kube-api-access-sprq9\") pod \"network-metrics-daemon-r5jrm\" (UID: \"e673ec85-406b-4221-8e0a-441d1886d20d\") " pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:35.375972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375339 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfn6t\" (UniqueName: \"kubernetes.io/projected/47cb0898-a067-4eb1-afe9-e2b5413b2cee-kube-api-access-gfn6t\") pod \"node-resolver-tvjd5\" (UID: \"47cb0898-a067-4eb1-afe9-e2b5413b2cee\") " pod="openshift-dns/node-resolver-tvjd5" Jul 09 18:25:35.375972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375373 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-sysctl-d\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.375972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375379 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-sys-fs\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.375972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375407 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-run-netns\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.375972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375440 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjnx2\" (UniqueName: \"kubernetes.io/projected/c6eea972-8391-4ba7-b1ce-9de56782baf3-kube-api-access-sjnx2\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.376802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375480 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-etc-openvswitch\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.376802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375519 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jr9d\" (UniqueName: \"kubernetes.io/projected/03116b04-5653-4707-8b77-a6c937981f49-kube-api-access-8jr9d\") pod \"node-ca-4b29x\" (UID: \"03116b04-5653-4707-8b77-a6c937981f49\") " pod="openshift-image-registry/node-ca-4b29x" Jul 09 18:25:35.376802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375546 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:35.376802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375577 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-socket-dir\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.376802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375618 2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 18:25:35.376802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375736 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-lib-modules\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.376802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375765 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-os-release\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.376802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375817 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-slash\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.376802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375844 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-ovn-node-metrics-cert\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.376802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375872 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03116b04-5653-4707-8b77-a6c937981f49-host\") pod \"node-ca-4b29x\" (UID: \"03116b04-5653-4707-8b77-a6c937981f49\") " pod="openshift-image-registry/node-ca-4b29x" Jul 09 18:25:35.376802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375898 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a93812e1-6915-47c8-b58a-d28334ab03a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.376802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375925 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftwgd\" (UniqueName: \"kubernetes.io/projected/52b201b5-436c-4b96-8c61-f519666aed5e-kube-api-access-ftwgd\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.376802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.375975 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-sysconfig\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.376802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376002 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-host\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.377536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376072 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-host\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.377536 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.376102 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:25:35.377536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376131 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-systemd-units\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.377536 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.376254 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs podName:e673ec85-406b-4221-8e0a-441d1886d20d nodeName:}" failed. No retries permitted until 2026-07-09 18:25:35.876238199 +0000 UTC m=+2.940903875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs") pod "network-metrics-daemon-r5jrm" (UID: "e673ec85-406b-4221-8e0a-441d1886d20d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:25:35.377536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376393 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a93812e1-6915-47c8-b58a-d28334ab03a3-system-cni-dir\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.377536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376431 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-registration-dir\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.377536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376443 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-sysconfig\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.377536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376463 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f352f25-0ced-43ed-bac7-7579314f9f89-agent-certs\") pod \"konnectivity-agent-9wfz4\" (UID: \"5f352f25-0ced-43ed-bac7-7579314f9f89\") " pod="kube-system/konnectivity-agent-9wfz4" Jul 09 18:25:35.377536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376493 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-multus-socket-dir-parent\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.377536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376521 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-var-lib-cni-bin\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.377536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376547 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6eea972-8391-4ba7-b1ce-9de56782baf3-multus-daemon-config\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.377536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376575 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-ovnkube-script-lib\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.377536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376580 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-lib-modules\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.377536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376615 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-kubelet-config\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:35.378291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376648 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-cni-netd\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.378291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376745 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-modprobe-d\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.378291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376792 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-systemd\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.378291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376822 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-system-cni-dir\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.378291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376875 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-run-netns\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.378291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376903 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a93812e1-6915-47c8-b58a-d28334ab03a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.378291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376932 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a93812e1-6915-47c8-b58a-d28334ab03a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.378291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.376962 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-sysctl-conf\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.378291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377034 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-run-systemd\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.378291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377062 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f352f25-0ced-43ed-bac7-7579314f9f89-konnectivity-ca\") pod \"konnectivity-agent-9wfz4\" (UID: \"5f352f25-0ced-43ed-bac7-7579314f9f89\") " pod="kube-system/konnectivity-agent-9wfz4" Jul 09 18:25:35.378291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377070 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-systemd\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.378291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377074 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-var-lib-openvswitch\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.378291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377199 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-modprobe-d\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.378291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377198 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-run-openvswitch\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.379013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377215 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-sysctl-conf\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.379013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377247 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.379013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377311 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb8zp\" (UniqueName: \"kubernetes.io/projected/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-kube-api-access-xb8zp\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.379013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377356 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.379013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377393 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-device-dir\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.379013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377435 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv8l4\" (UniqueName: \"kubernetes.io/projected/8f59262e-fc1c-4f6d-976d-228bc6cfe761-kube-api-access-wv8l4\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.379013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377484 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-var-lib-cni-multus\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.379013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377617 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-cni-bin\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.379013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377654 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.379013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377776 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-kubernetes\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.379013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377812 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-run\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.379013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.377862 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-var-lib-kubelet\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.379013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.378003 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-var-lib-kubelet\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.379013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.378102 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-run\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.379749 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.378156 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-kubernetes\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.380025 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.380000 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f352f25-0ced-43ed-bac7-7579314f9f89-agent-certs\") pod \"konnectivity-agent-9wfz4\" (UID: \"5f352f25-0ced-43ed-bac7-7579314f9f89\") " pod="kube-system/konnectivity-agent-9wfz4" Jul 09 18:25:35.381605 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.381575 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f59262e-fc1c-4f6d-976d-228bc6cfe761-tmp\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.382689 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.382645 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f59262e-fc1c-4f6d-976d-228bc6cfe761-etc-tuned\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.383643 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.383621 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:25:35.383737 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.383650 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:25:35.383737 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.383665 2558 projected.go:196] Error preparing data for projected volume kube-api-access-p6fxn for pod openshift-network-diagnostics/network-check-target-j2xsk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:25:35.383737 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.383723 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn podName:0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e nodeName:}" failed. No retries permitted until 2026-07-09 18:25:35.883710014 +0000 UTC m=+2.948375685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p6fxn" (UniqueName: "kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn") pod "network-check-target-j2xsk" (UID: "0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:25:35.394653 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.394627 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv8l4\" (UniqueName: \"kubernetes.io/projected/8f59262e-fc1c-4f6d-976d-228bc6cfe761-kube-api-access-wv8l4\") pod \"tuned-vnnp7\" (UID: \"8f59262e-fc1c-4f6d-976d-228bc6cfe761\") " pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.396491 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.396466 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprq9\" (UniqueName: \"kubernetes.io/projected/e673ec85-406b-4221-8e0a-441d1886d20d-kube-api-access-sprq9\") pod \"network-metrics-daemon-r5jrm\" (UID: \"e673ec85-406b-4221-8e0a-441d1886d20d\") " pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.478454 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a93812e1-6915-47c8-b58a-d28334ab03a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.478499 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x65k4\" (UniqueName: \"kubernetes.io/projected/a93812e1-6915-47c8-b58a-d28334ab03a3-kube-api-access-x65k4\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.478534 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47cb0898-a067-4eb1-afe9-e2b5413b2cee-tmp-dir\") pod \"node-resolver-tvjd5\" (UID: \"47cb0898-a067-4eb1-afe9-e2b5413b2cee\") " pod="openshift-dns/node-resolver-tvjd5" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.478559 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-multus-conf-dir\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.478587 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-etc-kubernetes\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.478610 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-run-ovn\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.478636 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-log-socket\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.478662 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.478689 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9256aa17-feeb-4e97-a4cc-af4488848183-iptables-alerter-script\") pod \"iptables-alerter-lfpjv\" (UID: \"9256aa17-feeb-4e97-a4cc-af4488848183\") " pod="openshift-network-operator/iptables-alerter-lfpjv" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.478737 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47cb0898-a067-4eb1-afe9-e2b5413b2cee-hosts-file\") pod \"node-resolver-tvjd5\" (UID: \"47cb0898-a067-4eb1-afe9-e2b5413b2cee\") " pod="openshift-dns/node-resolver-tvjd5" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.478828 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-etc-kubernetes\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.479015 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a93812e1-6915-47c8-b58a-d28334ab03a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.479558 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47cb0898-a067-4eb1-afe9-e2b5413b2cee-tmp-dir\") pod \"node-resolver-tvjd5\" (UID: \"47cb0898-a067-4eb1-afe9-e2b5413b2cee\") " pod="openshift-dns/node-resolver-tvjd5" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.479618 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-multus-conf-dir\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.479785 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.479669 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.480596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.479714 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-run-ovn\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.480596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.478762 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-var-lib-kubelet\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.480596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480280 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-node-log\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.480596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480324 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a93812e1-6915-47c8-b58a-d28334ab03a3-os-release\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.480596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480365 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-ovnkube-config\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.480596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480390 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/03116b04-5653-4707-8b77-a6c937981f49-serviceca\") pod \"node-ca-4b29x\" (UID: \"03116b04-5653-4707-8b77-a6c937981f49\") " pod="openshift-image-registry/node-ca-4b29x" Jul 09 18:25:35.480596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480413 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9256aa17-feeb-4e97-a4cc-af4488848183-host-slash\") pod \"iptables-alerter-lfpjv\" (UID: \"9256aa17-feeb-4e97-a4cc-af4488848183\") " pod="openshift-network-operator/iptables-alerter-lfpjv" Jul 09 18:25:35.480596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480438 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-cnibin\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.480596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480472 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-hostroot\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.480596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480495 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-run-multus-certs\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.480596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480522 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a93812e1-6915-47c8-b58a-d28334ab03a3-cnibin\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.480596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480549 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfn6t\" (UniqueName: \"kubernetes.io/projected/47cb0898-a067-4eb1-afe9-e2b5413b2cee-kube-api-access-gfn6t\") pod \"node-resolver-tvjd5\" (UID: \"47cb0898-a067-4eb1-afe9-e2b5413b2cee\") " pod="openshift-dns/node-resolver-tvjd5" Jul 09 18:25:35.480596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480573 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-sys-fs\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.480596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480599 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-run-netns\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.481309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480625 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjnx2\" (UniqueName: \"kubernetes.io/projected/c6eea972-8391-4ba7-b1ce-9de56782baf3-kube-api-access-sjnx2\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.481309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480667 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-etc-openvswitch\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.481309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480683 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jr9d\" (UniqueName: \"kubernetes.io/projected/03116b04-5653-4707-8b77-a6c937981f49-kube-api-access-8jr9d\") pod \"node-ca-4b29x\" (UID: \"03116b04-5653-4707-8b77-a6c937981f49\") " pod="openshift-image-registry/node-ca-4b29x" Jul 09 18:25:35.481309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480699 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:35.481309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480718 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-socket-dir\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.481309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480758 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-os-release\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.481309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480774 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-slash\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.481309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480790 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-ovn-node-metrics-cert\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.481309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480809 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03116b04-5653-4707-8b77-a6c937981f49-host\") pod \"node-ca-4b29x\" (UID: \"03116b04-5653-4707-8b77-a6c937981f49\") " pod="openshift-image-registry/node-ca-4b29x" Jul 09 18:25:35.481309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480822 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9256aa17-feeb-4e97-a4cc-af4488848183-iptables-alerter-script\") pod \"iptables-alerter-lfpjv\" (UID: \"9256aa17-feeb-4e97-a4cc-af4488848183\") " pod="openshift-network-operator/iptables-alerter-lfpjv" Jul 09 18:25:35.481309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480834 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a93812e1-6915-47c8-b58a-d28334ab03a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.481309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.480855 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwgd\" (UniqueName: \"kubernetes.io/projected/52b201b5-436c-4b96-8c61-f519666aed5e-kube-api-access-ftwgd\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.481309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481031 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-sys-fs\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.481309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481066 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-run-netns\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481138 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-var-lib-kubelet\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.479746 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-log-socket\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481205 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47cb0898-a067-4eb1-afe9-e2b5413b2cee-hosts-file\") pod \"node-resolver-tvjd5\" (UID: \"47cb0898-a067-4eb1-afe9-e2b5413b2cee\") " pod="openshift-dns/node-resolver-tvjd5" Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481249 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-etc-openvswitch\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481260 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-node-log\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481327 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a93812e1-6915-47c8-b58a-d28334ab03a3-os-release\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.481449 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.481496 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret podName:8cfe536a-207f-4ce7-8e2a-6a68d85fc813 nodeName:}" failed. No retries permitted until 2026-07-09 18:25:35.981484186 +0000 UTC m=+3.046149840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret") pod "global-pull-secret-syncer-r7g5z" (UID: "8cfe536a-207f-4ce7-8e2a-6a68d85fc813") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481613 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-socket-dir\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481663 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-hostroot\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481683 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-os-release\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481729 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-slash\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481789 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03116b04-5653-4707-8b77-a6c937981f49-host\") pod \"node-ca-4b29x\" (UID: \"03116b04-5653-4707-8b77-a6c937981f49\") " pod="openshift-image-registry/node-ca-4b29x" Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481795 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-run-multus-certs\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.481998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481887 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9256aa17-feeb-4e97-a4cc-af4488848183-host-slash\") pod \"iptables-alerter-lfpjv\" (UID: \"9256aa17-feeb-4e97-a4cc-af4488848183\") " pod="openshift-network-operator/iptables-alerter-lfpjv" Jul 09 18:25:35.482726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481928 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-systemd-units\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.482726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481957 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a93812e1-6915-47c8-b58a-d28334ab03a3-system-cni-dir\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.482726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.481985 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-registration-dir\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.482726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482011 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-multus-socket-dir-parent\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.482726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482036 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-var-lib-cni-bin\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.482726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482062 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6eea972-8391-4ba7-b1ce-9de56782baf3-multus-daemon-config\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.482726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482093 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-ovnkube-script-lib\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.482726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482121 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-kubelet-config\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:35.482726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482148 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-cni-netd\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.482726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482177 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-system-cni-dir\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.482726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482201 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-run-netns\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.482726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482227 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a93812e1-6915-47c8-b58a-d28334ab03a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.482726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482255 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a93812e1-6915-47c8-b58a-d28334ab03a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.482726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482282 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-run-systemd\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.484871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482286 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a93812e1-6915-47c8-b58a-d28334ab03a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.484871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482307 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-var-lib-openvswitch\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.484871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482333 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-run-openvswitch\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.484871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482375 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-run-openvswitch\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.484871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482379 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.484871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482415 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb8zp\" (UniqueName: \"kubernetes.io/projected/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-kube-api-access-xb8zp\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.484871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482431 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-kubelet-config\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:35.484871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482442 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.484871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482480 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-cni-netd\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.484871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482486 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-device-dir\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.484871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482533 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-system-cni-dir\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.484871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482551 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-device-dir\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.484871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482562 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-var-lib-cni-multus\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.484871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482592 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-cni-bin\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.485594 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482622 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.485594 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482633 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-cnibin\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.485594 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482657 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-run-k8s-cni-cncf-io\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.485594 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482685 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-kubelet\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.485594 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482710 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-registration-dir\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.485594 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482714 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8k9q\" (UniqueName: \"kubernetes.io/projected/9256aa17-feeb-4e97-a4cc-af4488848183-kube-api-access-k8k9q\") pod \"iptables-alerter-lfpjv\" (UID: \"9256aa17-feeb-4e97-a4cc-af4488848183\") " pod="openshift-network-operator/iptables-alerter-lfpjv" Jul 09 18:25:35.485594 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482780 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-multus-cni-dir\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.485594 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482821 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6eea972-8391-4ba7-b1ce-9de56782baf3-cni-binary-copy\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.485594 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482834 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.485594 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482857 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-env-overrides\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.485594 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482881 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-var-lib-cni-bin\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.485594 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482888 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-dbus\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:35.485594 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482882 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-var-lib-cni-multus\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.485594 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482591 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.482809 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-ovnkube-script-lib\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483024 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-multus-socket-dir-parent\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483056 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-var-lib-openvswitch\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483069 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-systemd-units\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483101 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-host-run-k8s-cni-cncf-io\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483114 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a93812e1-6915-47c8-b58a-d28334ab03a3-system-cni-dir\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483157 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-cni-bin\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483271 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52b201b5-436c-4b96-8c61-f519666aed5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483384 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a93812e1-6915-47c8-b58a-d28334ab03a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483391 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-run-systemd\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483435 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-kubelet\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483540 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6eea972-8391-4ba7-b1ce-9de56782baf3-multus-cni-dir\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483562 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-dbus\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483608 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-host-run-netns\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.486318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483692 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-env-overrides\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.487084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483693 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a93812e1-6915-47c8-b58a-d28334ab03a3-cnibin\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.487084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.483792 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6eea972-8391-4ba7-b1ce-9de56782baf3-cni-binary-copy\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.487084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.484064 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/03116b04-5653-4707-8b77-a6c937981f49-serviceca\") pod \"node-ca-4b29x\" (UID: \"03116b04-5653-4707-8b77-a6c937981f49\") " pod="openshift-image-registry/node-ca-4b29x" Jul 09 18:25:35.487084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.484078 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6eea972-8391-4ba7-b1ce-9de56782baf3-multus-daemon-config\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.487084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.484137 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a93812e1-6915-47c8-b58a-d28334ab03a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.487084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.484446 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-ovnkube-config\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.487647 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.487621 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-ovn-node-metrics-cert\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.500009 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.499944 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x65k4\" (UniqueName: \"kubernetes.io/projected/a93812e1-6915-47c8-b58a-d28334ab03a3-kube-api-access-x65k4\") pod \"multus-additional-cni-plugins-qtt72\" (UID: \"a93812e1-6915-47c8-b58a-d28334ab03a3\") " pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.506323 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.506249 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb8zp\" (UniqueName: \"kubernetes.io/projected/a41b928a-0e30-4d02-9d2c-0a6a4749d4f8-kube-api-access-xb8zp\") pod \"ovnkube-node-bmxck\" (UID: \"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.506445 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.506411 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8k9q\" (UniqueName: \"kubernetes.io/projected/9256aa17-feeb-4e97-a4cc-af4488848183-kube-api-access-k8k9q\") pod \"iptables-alerter-lfpjv\" (UID: \"9256aa17-feeb-4e97-a4cc-af4488848183\") " pod="openshift-network-operator/iptables-alerter-lfpjv" Jul 09 18:25:35.506908 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.506879 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jr9d\" (UniqueName: \"kubernetes.io/projected/03116b04-5653-4707-8b77-a6c937981f49-kube-api-access-8jr9d\") pod \"node-ca-4b29x\" (UID: \"03116b04-5653-4707-8b77-a6c937981f49\") " pod="openshift-image-registry/node-ca-4b29x" Jul 09 18:25:35.507126 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.507099 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfn6t\" (UniqueName: \"kubernetes.io/projected/47cb0898-a067-4eb1-afe9-e2b5413b2cee-kube-api-access-gfn6t\") pod \"node-resolver-tvjd5\" (UID: \"47cb0898-a067-4eb1-afe9-e2b5413b2cee\") " pod="openshift-dns/node-resolver-tvjd5" Jul 09 18:25:35.509806 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.509710 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftwgd\" (UniqueName: \"kubernetes.io/projected/52b201b5-436c-4b96-8c61-f519666aed5e-kube-api-access-ftwgd\") pod \"aws-ebs-csi-driver-node-qmq88\" (UID: \"52b201b5-436c-4b96-8c61-f519666aed5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.511548 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.511503 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjnx2\" (UniqueName: \"kubernetes.io/projected/c6eea972-8391-4ba7-b1ce-9de56782baf3-kube-api-access-sjnx2\") pod \"multus-dzbx9\" (UID: \"c6eea972-8391-4ba7-b1ce-9de56782baf3\") " pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.561131 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.561046 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal" event={"ID":"2c6925309b7197f6ab1e3cdd371df9a5","Type":"ContainerStarted","Data":"dbf81ba0b4327abc9c96371e03a0333586a95b1f17ee56f38dbcf0b5795fe1fc"} Jul 09 18:25:35.562505 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.562474 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-253.ec2.internal" event={"ID":"8c5edb6b0989376dda6e5d5712ab11e9","Type":"ContainerStarted","Data":"ec038d7c1b551cc7420a15d3a974ddd1f293a44f0f6cc6badbce2a19c8a60fd1"} Jul 09 18:25:35.571139 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.570918 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" Jul 09 18:25:35.578416 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.578397 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9wfz4" Jul 09 18:25:35.582949 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:35.582915 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f59262e_fc1c_4f6d_976d_228bc6cfe761.slice/crio-b336631d3651128b40f09734757709a20fdee7471435bcd817e9dbc7e6a03792 WatchSource:0}: Error finding container b336631d3651128b40f09734757709a20fdee7471435bcd817e9dbc7e6a03792: Status 404 returned error can't find the container with id b336631d3651128b40f09734757709a20fdee7471435bcd817e9dbc7e6a03792 Jul 09 18:25:35.589288 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:35.589261 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f352f25_0ced_43ed_bac7_7579314f9f89.slice/crio-2b85dc4f187c6622b58baf715c15b7430029629c4f1194b538be466ee538c75b WatchSource:0}: Error finding container 2b85dc4f187c6622b58baf715c15b7430029629c4f1194b538be466ee538c75b: Status 404 returned error can't find the container with id 2b85dc4f187c6622b58baf715c15b7430029629c4f1194b538be466ee538c75b Jul 09 18:25:35.589401 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.589334 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tvjd5" Jul 09 18:25:35.597534 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.597505 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qtt72" Jul 09 18:25:35.600945 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:35.600078 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cb0898_a067_4eb1_afe9_e2b5413b2cee.slice/crio-88d413bfa448d098928c3dc83043f953374242e08f3ffb7da67a440e3590c20a WatchSource:0}: Error finding container 88d413bfa448d098928c3dc83043f953374242e08f3ffb7da67a440e3590c20a: Status 404 returned error can't find the container with id 88d413bfa448d098928c3dc83043f953374242e08f3ffb7da67a440e3590c20a Jul 09 18:25:35.606972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.606816 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" Jul 09 18:25:35.607516 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:35.607491 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93812e1_6915_47c8_b58a_d28334ab03a3.slice/crio-2589c805b3ff3670159f9250895248d3f0989ccc7ee6590924fdb845b3d74b79 WatchSource:0}: Error finding container 2589c805b3ff3670159f9250895248d3f0989ccc7ee6590924fdb845b3d74b79: Status 404 returned error can't find the container with id 2589c805b3ff3670159f9250895248d3f0989ccc7ee6590924fdb845b3d74b79 Jul 09 18:25:35.616487 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.616327 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dzbx9" Jul 09 18:25:35.625870 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:35.625845 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6eea972_8391_4ba7_b1ce_9de56782baf3.slice/crio-2784a652d8bb5e2f7a5898a39d5d43e7ee72bf19a2b9fdfb90c2ce5eb8377aee WatchSource:0}: Error finding container 2784a652d8bb5e2f7a5898a39d5d43e7ee72bf19a2b9fdfb90c2ce5eb8377aee: Status 404 returned error can't find the container with id 2784a652d8bb5e2f7a5898a39d5d43e7ee72bf19a2b9fdfb90c2ce5eb8377aee Jul 09 18:25:35.626652 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.626632 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:25:35.635955 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.635934 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lfpjv" Jul 09 18:25:35.636454 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:35.636426 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41b928a_0e30_4d02_9d2c_0a6a4749d4f8.slice/crio-3efe6bd174e9f40751dfde014b5f3f070fbdfae65ce7589d80d11bd16a6153c8 WatchSource:0}: Error finding container 3efe6bd174e9f40751dfde014b5f3f070fbdfae65ce7589d80d11bd16a6153c8: Status 404 returned error can't find the container with id 3efe6bd174e9f40751dfde014b5f3f070fbdfae65ce7589d80d11bd16a6153c8 Jul 09 18:25:35.645089 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.645064 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4b29x" Jul 09 18:25:35.647840 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:35.647813 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9256aa17_feeb_4e97_a4cc_af4488848183.slice/crio-226d8112b6355d6d6d0cdfc42f20300db4f1803caf9337852e2ceee59e43204f WatchSource:0}: Error finding container 226d8112b6355d6d6d0cdfc42f20300db4f1803caf9337852e2ceee59e43204f: Status 404 returned error can't find the container with id 226d8112b6355d6d6d0cdfc42f20300db4f1803caf9337852e2ceee59e43204f Jul 09 18:25:35.660038 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:25:35.660008 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03116b04_5653_4707_8b77_a6c937981f49.slice/crio-23ab29b35227b1205717e9b0d059fc85c222a0cbaade90f61beeba32fb39ba68 WatchSource:0}: Error finding container 23ab29b35227b1205717e9b0d059fc85c222a0cbaade90f61beeba32fb39ba68: Status 404 returned error can't find the container with id 23ab29b35227b1205717e9b0d059fc85c222a0cbaade90f61beeba32fb39ba68 Jul 09 18:25:35.885574 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.885489 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fxn\" (UniqueName: \"kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn\") pod \"network-check-target-j2xsk\" (UID: \"0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e\") " pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:35.885574 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.885529 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs\") pod \"network-metrics-daemon-r5jrm\" (UID: \"e673ec85-406b-4221-8e0a-441d1886d20d\") " pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:35.885817 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.885721 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:25:35.885817 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.885748 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:25:35.885817 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.885756 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:25:35.885942 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.885834 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs podName:e673ec85-406b-4221-8e0a-441d1886d20d nodeName:}" failed. No retries permitted until 2026-07-09 18:25:36.885800906 +0000 UTC m=+3.950466559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs") pod "network-metrics-daemon-r5jrm" (UID: "e673ec85-406b-4221-8e0a-441d1886d20d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:25:35.885942 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.885763 2558 projected.go:196] Error preparing data for projected volume kube-api-access-p6fxn for pod openshift-network-diagnostics/network-check-target-j2xsk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:25:35.885942 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.885933 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn podName:0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e nodeName:}" failed. No retries permitted until 2026-07-09 18:25:36.885916925 +0000 UTC m=+3.950582595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6fxn" (UniqueName: "kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn") pod "network-check-target-j2xsk" (UID: "0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:25:35.986716 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:35.986673 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:35.986880 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.986852 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:25:35.986950 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:35.986938 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret podName:8cfe536a-207f-4ce7-8e2a-6a68d85fc813 nodeName:}" failed. No retries permitted until 2026-07-09 18:25:36.986916493 +0000 UTC m=+4.051582168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret") pod "global-pull-secret-syncer-r7g5z" (UID: "8cfe536a-207f-4ce7-8e2a-6a68d85fc813") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:25:36.373107 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.373014 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 18:20:35 +0000 UTC" deadline="2027-12-11 14:27:05.042004469 +0000 UTC" Jul 09 18:25:36.373107 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.373060 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12476h1m28.668949677s" Jul 09 18:25:36.553754 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.553720 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:36.554368 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:36.553848 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:25:36.566000 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.565966 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtt72" event={"ID":"a93812e1-6915-47c8-b58a-d28334ab03a3","Type":"ContainerStarted","Data":"2589c805b3ff3670159f9250895248d3f0989ccc7ee6590924fdb845b3d74b79"} Jul 09 18:25:36.567274 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.567206 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tvjd5" event={"ID":"47cb0898-a067-4eb1-afe9-e2b5413b2cee","Type":"ContainerStarted","Data":"88d413bfa448d098928c3dc83043f953374242e08f3ffb7da67a440e3590c20a"} Jul 09 18:25:36.569598 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.569525 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9wfz4" event={"ID":"5f352f25-0ced-43ed-bac7-7579314f9f89","Type":"ContainerStarted","Data":"2b85dc4f187c6622b58baf715c15b7430029629c4f1194b538be466ee538c75b"} Jul 09 18:25:36.571892 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.571832 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" event={"ID":"8f59262e-fc1c-4f6d-976d-228bc6cfe761","Type":"ContainerStarted","Data":"b336631d3651128b40f09734757709a20fdee7471435bcd817e9dbc7e6a03792"} Jul 09 18:25:36.573412 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.573338 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4b29x" event={"ID":"03116b04-5653-4707-8b77-a6c937981f49","Type":"ContainerStarted","Data":"23ab29b35227b1205717e9b0d059fc85c222a0cbaade90f61beeba32fb39ba68"} Jul 09 18:25:36.575241 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.575168 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lfpjv" event={"ID":"9256aa17-feeb-4e97-a4cc-af4488848183","Type":"ContainerStarted","Data":"226d8112b6355d6d6d0cdfc42f20300db4f1803caf9337852e2ceee59e43204f"} Jul 09 18:25:36.577644 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.577543 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" event={"ID":"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8","Type":"ContainerStarted","Data":"3efe6bd174e9f40751dfde014b5f3f070fbdfae65ce7589d80d11bd16a6153c8"} Jul 09 18:25:36.578923 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.578903 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dzbx9" event={"ID":"c6eea972-8391-4ba7-b1ce-9de56782baf3","Type":"ContainerStarted","Data":"2784a652d8bb5e2f7a5898a39d5d43e7ee72bf19a2b9fdfb90c2ce5eb8377aee"} Jul 09 18:25:36.580714 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.580679 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" event={"ID":"52b201b5-436c-4b96-8c61-f519666aed5e","Type":"ContainerStarted","Data":"3c17c60c68bddbd76c3425ccedc46215c0131937a84b454cff20be1f07a3a1c5"} Jul 09 18:25:36.898531 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.897802 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fxn\" (UniqueName: \"kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn\") pod \"network-check-target-j2xsk\" (UID: \"0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e\") " pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:36.898531 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.897862 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs\") pod \"network-metrics-daemon-r5jrm\" (UID: \"e673ec85-406b-4221-8e0a-441d1886d20d\") " pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:36.898531 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:36.898078 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:25:36.898531 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:36.898144 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs podName:e673ec85-406b-4221-8e0a-441d1886d20d nodeName:}" failed. No retries permitted until 2026-07-09 18:25:38.8981252 +0000 UTC m=+5.962790876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs") pod "network-metrics-daemon-r5jrm" (UID: "e673ec85-406b-4221-8e0a-441d1886d20d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:25:36.898531 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:36.898340 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:25:36.898531 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:36.898359 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:25:36.898531 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:36.898372 2558 projected.go:196] Error preparing data for projected volume kube-api-access-p6fxn for pod openshift-network-diagnostics/network-check-target-j2xsk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:25:36.898531 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:36.898528 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn podName:0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e nodeName:}" failed. No retries permitted until 2026-07-09 18:25:38.898509283 +0000 UTC m=+5.963174953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6fxn" (UniqueName: "kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn") pod "network-check-target-j2xsk" (UID: "0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:25:36.999055 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:36.999015 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:36.999240 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:36.999213 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:25:36.999296 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:36.999276 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret podName:8cfe536a-207f-4ce7-8e2a-6a68d85fc813 nodeName:}" failed. No retries permitted until 2026-07-09 18:25:38.999259445 +0000 UTC m=+6.063925101 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret") pod "global-pull-secret-syncer-r7g5z" (UID: "8cfe536a-207f-4ce7-8e2a-6a68d85fc813") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:25:37.314438 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:37.314144 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:25:37.378720 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:37.373835 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 18:20:35 +0000 UTC" deadline="2028-04-22 11:15:06.50668966 +0000 UTC" Jul 09 18:25:37.378720 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:37.374207 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15664h49m29.132675434s" Jul 09 18:25:37.556582 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:37.555906 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:37.556582 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:37.556045 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:25:37.556582 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:37.556120 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:37.556582 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:37.556189 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:25:38.553883 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:38.553314 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:38.553883 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:38.553768 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:25:38.915256 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:38.914391 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fxn\" (UniqueName: \"kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn\") pod \"network-check-target-j2xsk\" (UID: \"0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e\") " pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:38.915256 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:38.914448 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs\") pod \"network-metrics-daemon-r5jrm\" (UID: \"e673ec85-406b-4221-8e0a-441d1886d20d\") " pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:38.915256 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:38.914608 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:25:38.915256 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:38.914680 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs podName:e673ec85-406b-4221-8e0a-441d1886d20d nodeName:}" failed. No retries permitted until 2026-07-09 18:25:42.914660667 +0000 UTC m=+9.979326321 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs") pod "network-metrics-daemon-r5jrm" (UID: "e673ec85-406b-4221-8e0a-441d1886d20d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:25:38.915256 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:38.915096 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:25:38.915256 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:38.915115 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:25:38.915256 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:38.915128 2558 projected.go:196] Error preparing data for projected volume kube-api-access-p6fxn for pod openshift-network-diagnostics/network-check-target-j2xsk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:25:38.915256 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:38.915172 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn podName:0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e nodeName:}" failed. No retries permitted until 2026-07-09 18:25:42.915156965 +0000 UTC m=+9.979822623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6fxn" (UniqueName: "kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn") pod "network-check-target-j2xsk" (UID: "0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:25:39.015389 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:39.015349 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:39.015556 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:39.015546 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:25:39.015627 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:39.015609 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret podName:8cfe536a-207f-4ce7-8e2a-6a68d85fc813 nodeName:}" failed. No retries permitted until 2026-07-09 18:25:43.015591468 +0000 UTC m=+10.080257143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret") pod "global-pull-secret-syncer-r7g5z" (UID: "8cfe536a-207f-4ce7-8e2a-6a68d85fc813") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:25:39.555059 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:39.555026 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:39.555226 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:39.555161 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:25:39.555284 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:39.555195 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:39.555406 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:39.555380 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:25:40.554138 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:40.554102 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:40.554588 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:40.554235 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:25:41.554096 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:41.554019 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:41.554257 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:41.554155 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:25:41.554606 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:41.554560 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:41.554690 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:41.554669 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:25:42.554321 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:42.554282 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:42.554837 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:42.554422 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:25:42.945430 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:42.945340 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fxn\" (UniqueName: \"kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn\") pod \"network-check-target-j2xsk\" (UID: \"0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e\") " pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:42.945430 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:42.945387 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs\") pod \"network-metrics-daemon-r5jrm\" (UID: \"e673ec85-406b-4221-8e0a-441d1886d20d\") " pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:42.945633 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:42.945523 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:25:42.945633 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:42.945581 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs podName:e673ec85-406b-4221-8e0a-441d1886d20d nodeName:}" failed. No retries permitted until 2026-07-09 18:25:50.945562552 +0000 UTC m=+18.010228204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs") pod "network-metrics-daemon-r5jrm" (UID: "e673ec85-406b-4221-8e0a-441d1886d20d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:25:42.945906 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:42.945882 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:25:42.945906 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:42.945908 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:25:42.946094 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:42.945934 2558 projected.go:196] Error preparing data for projected volume kube-api-access-p6fxn for pod openshift-network-diagnostics/network-check-target-j2xsk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:25:42.946094 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:42.945986 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn podName:0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e nodeName:}" failed. No retries permitted until 2026-07-09 18:25:50.945973096 +0000 UTC m=+18.010638752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6fxn" (UniqueName: "kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn") pod "network-check-target-j2xsk" (UID: "0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:25:43.047346 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:43.046770 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:43.047346 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:43.046937 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:25:43.047346 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:43.047003 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret podName:8cfe536a-207f-4ce7-8e2a-6a68d85fc813 nodeName:}" failed. No retries permitted until 2026-07-09 18:25:51.046982866 +0000 UTC m=+18.111648543 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret") pod "global-pull-secret-syncer-r7g5z" (UID: "8cfe536a-207f-4ce7-8e2a-6a68d85fc813") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:25:43.554319 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:43.554286 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:43.554483 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:43.554386 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:25:43.554851 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:43.554807 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:43.555019 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:43.554915 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:25:44.553880 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:44.553848 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:44.554070 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:44.553989 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:25:45.553277 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:45.553243 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:45.553277 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:45.553276 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:45.553770 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:45.553417 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:25:45.553770 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:45.553523 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:25:46.553833 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:46.553798 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:46.554230 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:46.553931 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:25:47.553504 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:47.553467 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:47.553690 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:47.553606 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:25:47.553690 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:47.553468 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:47.553818 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:47.553790 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:25:48.553773 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:48.553732 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:48.554221 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:48.553874 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:25:49.553715 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:49.553496 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:49.553909 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:49.553583 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:49.553909 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:49.553828 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:25:49.553909 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:49.553895 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:25:50.553509 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:50.553473 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:50.553688 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:50.553581 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:25:51.008306 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:51.008268 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fxn\" (UniqueName: \"kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn\") pod \"network-check-target-j2xsk\" (UID: \"0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e\") " pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:51.008306 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:51.008310 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs\") pod \"network-metrics-daemon-r5jrm\" (UID: \"e673ec85-406b-4221-8e0a-441d1886d20d\") " pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:51.008823 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:51.008399 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:25:51.008823 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:51.008431 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:25:51.008823 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:51.008454 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:25:51.008823 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:51.008462 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs podName:e673ec85-406b-4221-8e0a-441d1886d20d nodeName:}" failed. No retries permitted until 2026-07-09 18:26:07.008444394 +0000 UTC m=+34.073110049 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs") pod "network-metrics-daemon-r5jrm" (UID: "e673ec85-406b-4221-8e0a-441d1886d20d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:25:51.008823 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:51.008468 2558 projected.go:196] Error preparing data for projected volume kube-api-access-p6fxn for pod openshift-network-diagnostics/network-check-target-j2xsk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:25:51.008823 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:51.008515 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn podName:0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e nodeName:}" failed. No retries permitted until 2026-07-09 18:26:07.008500928 +0000 UTC m=+34.073166601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6fxn" (UniqueName: "kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn") pod "network-check-target-j2xsk" (UID: "0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:25:51.109238 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:51.109193 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:51.109398 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:51.109353 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:25:51.109460 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:51.109444 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret podName:8cfe536a-207f-4ce7-8e2a-6a68d85fc813 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:07.109419993 +0000 UTC m=+34.174085677 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret") pod "global-pull-secret-syncer-r7g5z" (UID: "8cfe536a-207f-4ce7-8e2a-6a68d85fc813") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:25:51.554299 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:51.554263 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:51.554299 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:51.554292 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:51.554579 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:51.554440 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:25:51.554636 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:51.554587 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:25:52.554202 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:52.554170 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:52.554605 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:52.554276 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:25:53.554054 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:53.554015 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:53.554252 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:53.554115 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:25:53.554252 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:53.554195 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:53.554650 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:53.554322 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:25:54.553992 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:54.553912 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:54.554123 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:54.554021 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:25:55.553749 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.553682 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:55.554360 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.553733 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:55.554360 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:55.553869 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:25:55.554360 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:55.553884 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:25:55.627019 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.626902 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dzbx9" event={"ID":"c6eea972-8391-4ba7-b1ce-9de56782baf3","Type":"ContainerStarted","Data":"ddd175837c7b8a61e2aa3626faa1fed2fd673c77aff1e4d2163b7707377874eb"} Jul 09 18:25:55.628361 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.628337 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" event={"ID":"52b201b5-436c-4b96-8c61-f519666aed5e","Type":"ContainerStarted","Data":"f42098bf2e42ba1c62d29240bca5d11c627b044c02ba8612a74d1b8feea98e39"} Jul 09 18:25:55.629644 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.629617 2558 generic.go:356] "Generic (PLEG): container finished" podID="a93812e1-6915-47c8-b58a-d28334ab03a3" containerID="c7f2c604726e690fc0cd6ca6c74f3af1a79e620634a2eac92647179f08e12d88" exitCode=0 Jul 09 18:25:55.629753 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.629718 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtt72" event={"ID":"a93812e1-6915-47c8-b58a-d28334ab03a3","Type":"ContainerDied","Data":"c7f2c604726e690fc0cd6ca6c74f3af1a79e620634a2eac92647179f08e12d88"} Jul 09 18:25:55.631119 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.631103 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tvjd5" event={"ID":"47cb0898-a067-4eb1-afe9-e2b5413b2cee","Type":"ContainerStarted","Data":"a6a05aeb337f0eeb6f8d003c4163c81b46b6d28a994de2106fbcfecb416f58f0"} Jul 09 18:25:55.632406 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.632385 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9wfz4" event={"ID":"5f352f25-0ced-43ed-bac7-7579314f9f89","Type":"ContainerStarted","Data":"5f7bcec54abc6dcac53ac1d610d710ccb9d0f135fd3e893c46f1fcc94027744b"} Jul 09 18:25:55.633863 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.633835 2558 generic.go:356] "Generic (PLEG): container finished" podID="2c6925309b7197f6ab1e3cdd371df9a5" containerID="b31f9561df828e7463233592455ab57f42c38512344d6e3961ab4f864481ae83" exitCode=0 Jul 09 18:25:55.633956 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.633899 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal" event={"ID":"2c6925309b7197f6ab1e3cdd371df9a5","Type":"ContainerDied","Data":"b31f9561df828e7463233592455ab57f42c38512344d6e3961ab4f864481ae83"} Jul 09 18:25:55.635341 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.635318 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-253.ec2.internal" event={"ID":"8c5edb6b0989376dda6e5d5712ab11e9","Type":"ContainerStarted","Data":"6141ad6cab7ba0c9b0df9f7ff4c36600d86f59710261be9cc26404db27806f37"} Jul 09 18:25:55.636873 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.636844 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" event={"ID":"8f59262e-fc1c-4f6d-976d-228bc6cfe761","Type":"ContainerStarted","Data":"4063429a1502d9f4de755ce31cdf55e6107eccf743f299bf4665177a7bf7d0dd"} Jul 09 18:25:55.638132 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.638115 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4b29x" event={"ID":"03116b04-5653-4707-8b77-a6c937981f49","Type":"ContainerStarted","Data":"a5d99a9380d0852e2255e5b5f72b8ffb02adc4a77cac25bf51ad7aaab0b9f3a6"} Jul 09 18:25:55.640649 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.640632 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" event={"ID":"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8","Type":"ContainerStarted","Data":"dc55432773d4401c7c31baffd760bad92467db8e53e85ed0f5e3ad9992b0f52b"} Jul 09 18:25:55.640735 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.640684 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" event={"ID":"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8","Type":"ContainerStarted","Data":"b838df54c65a86abe64f19c80ef5e9fea20614afbcd14d65fc05e329966d89c9"} Jul 09 18:25:55.640772 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.640735 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" event={"ID":"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8","Type":"ContainerStarted","Data":"ec243a6b4ac16db074e95cc7773306af5fb78b5b7ec79ede7685363ef5d64845"} Jul 09 18:25:55.640772 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.640763 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" event={"ID":"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8","Type":"ContainerStarted","Data":"9b62519745220c11ff859e90260401c853bc1bd8154764d7446a8277f149d3b9"} Jul 09 18:25:55.640834 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.640791 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" event={"ID":"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8","Type":"ContainerStarted","Data":"892e5a8c9d67fa76d5bbf23eecd5616e58b1999324e81adf85a4f3c0704318c9"} Jul 09 18:25:55.640834 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.640818 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" event={"ID":"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8","Type":"ContainerStarted","Data":"5c02761282ed1c810124f9d6b9361956656ad6e6443c4a3a86f3d954d42635c2"} Jul 09 18:25:55.694524 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.694470 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dzbx9" podStartSLOduration=3.585768009 podStartE2EDuration="22.694454129s" totalImagesPullingTime="19.10868612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:55.666847139 +0000 UTC m=+22.731512825" watchObservedRunningTime="2026-07-09 18:25:55.694454129 +0000 UTC m=+22.759119803" Jul 09 18:25:55.694888 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.694859 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4b29x" podStartSLOduration=4.149204052 podStartE2EDuration="22.694847285s" totalImagesPullingTime="18.545643233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:55.694287904 +0000 UTC m=+22.758953580" watchObservedRunningTime="2026-07-09 18:25:55.694847285 +0000 UTC m=+22.759512961" Jul 09 18:25:55.759883 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.759814 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-253.ec2.internal" podStartSLOduration=21.759792193 podStartE2EDuration="21.759792193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:55.752425369 +0000 UTC m=+22.817091046" watchObservedRunningTime="2026-07-09 18:25:55.759792193 +0000 UTC m=+22.824457868" Jul 09 18:25:55.803330 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.803284 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vnnp7" podStartSLOduration=3.732120239 podStartE2EDuration="22.803266669s" totalImagesPullingTime="19.07114643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:55.803107404 +0000 UTC m=+22.867773077" watchObservedRunningTime="2026-07-09 18:25:55.803266669 +0000 UTC m=+22.867932347" Jul 09 18:25:55.821136 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.821095 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9wfz4" podStartSLOduration=8.087641412 podStartE2EDuration="22.821078444s" totalImagesPullingTime="14.733437032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:55.820873299 +0000 UTC m=+22.885538973" watchObservedRunningTime="2026-07-09 18:25:55.821078444 +0000 UTC m=+22.885744120" Jul 09 18:25:55.838801 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:55.838763 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tvjd5" podStartSLOduration=3.786209254 podStartE2EDuration="22.838750921s" totalImagesPullingTime="19.052541667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:55.838664347 +0000 UTC m=+22.903330020" watchObservedRunningTime="2026-07-09 18:25:55.838750921 +0000 UTC m=+22.903416591" Jul 09 18:25:56.554011 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:56.553984 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:56.554349 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:56.554099 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:25:56.637463 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:56.637429 2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 18:25:56.644341 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:56.644318 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" event={"ID":"52b201b5-436c-4b96-8c61-f519666aed5e","Type":"ContainerStarted","Data":"3ea4a3b83a5210eb82939bee87fefed5c71e641c391bd53582c1c91e3c64b941"} Jul 09 18:25:56.646038 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:56.646008 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal" event={"ID":"2c6925309b7197f6ab1e3cdd371df9a5","Type":"ContainerStarted","Data":"35ef1d164028c5b67afb0c2d090ddbb5dd6c1abc969dc9d51c4e603766f100d5"} Jul 09 18:25:56.649414 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:56.649386 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lfpjv" event={"ID":"9256aa17-feeb-4e97-a4cc-af4488848183","Type":"ContainerStarted","Data":"bd994f063ba7fd767ccf810b33e7fabecdecd3a7bc1be55fb9b8067dc79db5ba"} Jul 09 18:25:56.664754 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:56.664694 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-253.ec2.internal" podStartSLOduration=22.664683696 podStartE2EDuration="22.664683696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:56.664392069 +0000 UTC m=+23.729057742" watchObservedRunningTime="2026-07-09 18:25:56.664683696 +0000 UTC m=+23.729349370" Jul 09 18:25:56.682331 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:56.682294 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lfpjv" podStartSLOduration=5.1251459520000004 podStartE2EDuration="23.682282076s" totalImagesPullingTime="18.557136124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:56.68164968 +0000 UTC m=+23.746315355" watchObservedRunningTime="2026-07-09 18:25:56.682282076 +0000 UTC m=+23.746947750" Jul 09 18:25:57.336860 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:57.336754 2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:25:56.637450163Z","UUID":"e6a2fd27-6bd4-476b-8438-c9a4c7cef9b4","Handler":null,"Name":"","Endpoint":""} Jul 09 18:25:57.338460 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:57.338424 2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 18:25:57.338614 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:57.338471 2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 18:25:57.554260 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:57.554231 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:57.554683 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:57.554334 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:25:57.554683 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:57.554386 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:57.554683 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:57.554476 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:25:57.655611 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:57.655575 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" event={"ID":"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8","Type":"ContainerStarted","Data":"def8352630a3fdba7875629f53de782d1cc0830de349c0655d2b44ca0c8ffd6e"} Jul 09 18:25:58.553544 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:58.553510 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:25:58.553734 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:58.553633 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:25:58.660738 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:58.660690 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" event={"ID":"52b201b5-436c-4b96-8c61-f519666aed5e","Type":"ContainerStarted","Data":"ec25c2c880a8759a38a41c477ff3afbb7ee396bffbcd4e54cf947bdd73de977e"} Jul 09 18:25:58.681866 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:58.681802 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmq88" podStartSLOduration=4.246661734 podStartE2EDuration="25.681786186s" totalImagesPullingTime="21.435124452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:25:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:58.681121939 +0000 UTC m=+25.745787613" watchObservedRunningTime="2026-07-09 18:25:58.681786186 +0000 UTC m=+25.746451860" Jul 09 18:25:58.717646 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:58.717611 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9wfz4" Jul 09 18:25:58.718211 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:58.718193 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9wfz4" Jul 09 18:25:59.554125 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:59.554086 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:25:59.554275 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:59.554200 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:25:59.554330 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:25:59.554284 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:25:59.554398 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:25:59.554380 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:26:00.554188 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:00.553942 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:26:00.554840 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:00.554287 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:26:00.669053 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:00.669018 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" event={"ID":"a41b928a-0e30-4d02-9d2c-0a6a4749d4f8","Type":"ContainerStarted","Data":"3521240e37f3d8f22776ddff9851e6d419d443deb9ab2442a659d6ac66b36fa7"} Jul 09 18:26:00.669343 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:00.669319 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:26:00.685779 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:00.685758 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:26:00.718048 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:00.718000 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" podStartSLOduration=8.680169776 podStartE2EDuration="27.717985624s" totalImagesPullingTime="19.037815848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:00.715214809 +0000 UTC m=+27.779880483" watchObservedRunningTime="2026-07-09 18:26:00.717985624 +0000 UTC m=+27.782651297" Jul 09 18:26:01.553808 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:01.553781 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:26:01.553808 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:01.553798 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:26:01.554012 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:01.553941 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:26:01.554092 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:01.554029 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:26:01.672651 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:01.672618 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:26:01.673242 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:01.672728 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:26:01.691985 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:01.691959 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:26:01.960386 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:01.960213 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r7g5z"] Jul 09 18:26:01.960548 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:01.960494 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:26:01.960597 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:01.960580 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:26:01.963015 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:01.962987 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r5jrm"] Jul 09 18:26:01.963152 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:01.963108 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:26:01.963243 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:01.963217 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:26:01.963600 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:01.963581 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j2xsk"] Jul 09 18:26:01.963715 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:01.963678 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:26:01.963815 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:01.963772 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:26:02.675382 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:02.675349 2558 generic.go:356] "Generic (PLEG): container finished" podID="a93812e1-6915-47c8-b58a-d28334ab03a3" containerID="128ba4abe50b57d82cc05fa0fd02617b7d9408d74d210f9a1f3d7cb12fe4541d" exitCode=0 Jul 09 18:26:02.675732 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:02.675438 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtt72" event={"ID":"a93812e1-6915-47c8-b58a-d28334ab03a3","Type":"ContainerDied","Data":"128ba4abe50b57d82cc05fa0fd02617b7d9408d74d210f9a1f3d7cb12fe4541d"} Jul 09 18:26:02.978378 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:02.978293 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9wfz4" Jul 09 18:26:02.978567 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:02.978399 2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 18:26:02.979032 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:02.979011 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9wfz4" Jul 09 18:26:03.554568 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:03.554534 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:26:03.554724 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:03.554623 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:26:03.554724 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:03.554708 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:26:03.554818 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:03.554800 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:26:03.554856 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:03.554843 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:26:03.554908 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:03.554896 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:26:04.681735 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:04.681537 2558 generic.go:356] "Generic (PLEG): container finished" podID="a93812e1-6915-47c8-b58a-d28334ab03a3" containerID="b05c922580994185e513dd6b5e8c28bc0c0c9a8d27e81a5a655cf8a5b610c1c3" exitCode=0 Jul 09 18:26:04.681735 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:04.681621 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtt72" event={"ID":"a93812e1-6915-47c8-b58a-d28334ab03a3","Type":"ContainerDied","Data":"b05c922580994185e513dd6b5e8c28bc0c0c9a8d27e81a5a655cf8a5b610c1c3"} Jul 09 18:26:04.699872 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:04.699839 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:04.780820 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:04.780784 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:05.553769 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:05.553744 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:26:05.553769 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:05.553752 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:26:05.553980 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:05.553840 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2xsk" podUID="0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e" Jul 09 18:26:05.553980 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:05.553897 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:26:05.554084 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:05.553991 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5jrm" podUID="e673ec85-406b-4221-8e0a-441d1886d20d" Jul 09 18:26:05.554084 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:05.554065 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7g5z" podUID="8cfe536a-207f-4ce7-8e2a-6a68d85fc813" Jul 09 18:26:05.685717 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:05.685672 2558 generic.go:356] "Generic (PLEG): container finished" podID="a93812e1-6915-47c8-b58a-d28334ab03a3" containerID="dd555b5ef1efc2aa6c63361a6f23d397830827c97b590b9a0411ddf1d07712cf" exitCode=0 Jul 09 18:26:05.686061 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:05.685732 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtt72" event={"ID":"a93812e1-6915-47c8-b58a-d28334ab03a3","Type":"ContainerDied","Data":"dd555b5ef1efc2aa6c63361a6f23d397830827c97b590b9a0411ddf1d07712cf"} Jul 09 18:26:07.040374 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.040341 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fxn\" (UniqueName: \"kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn\") pod \"network-check-target-j2xsk\" (UID: \"0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e\") " pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:26:07.040898 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.040381 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs\") pod \"network-metrics-daemon-r5jrm\" (UID: \"e673ec85-406b-4221-8e0a-441d1886d20d\") " pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:26:07.040898 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:07.040483 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:26:07.040898 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:07.040500 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:26:07.040898 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:07.040522 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:26:07.040898 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:07.040535 2558 projected.go:196] Error preparing data for projected volume kube-api-access-p6fxn for pod openshift-network-diagnostics/network-check-target-j2xsk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:26:07.040898 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:07.040561 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs podName:e673ec85-406b-4221-8e0a-441d1886d20d nodeName:}" failed. No retries permitted until 2026-07-09 18:26:39.0405413 +0000 UTC m=+66.105206969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs") pod "network-metrics-daemon-r5jrm" (UID: "e673ec85-406b-4221-8e0a-441d1886d20d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:26:07.040898 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:07.040580 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn podName:0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e nodeName:}" failed. No retries permitted until 2026-07-09 18:26:39.040570513 +0000 UTC m=+66.105236171 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6fxn" (UniqueName: "kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn") pod "network-check-target-j2xsk" (UID: "0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:26:07.140821 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.140788 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:26:07.140962 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:07.140933 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:26:07.141005 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:07.140991 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret podName:8cfe536a-207f-4ce7-8e2a-6a68d85fc813 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:39.140975504 +0000 UTC m=+66.205641162 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret") pod "global-pull-secret-syncer-r7g5z" (UID: "8cfe536a-207f-4ce7-8e2a-6a68d85fc813") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:26:07.331181 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.331107 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-253.ec2.internal" event="NodeReady" Jul 09 18:26:07.331329 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.331244 2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 18:26:07.387833 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.387799 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4txx9"] Jul 09 18:26:07.390738 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.390722 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-twfp7"] Jul 09 18:26:07.390842 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.390825 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:07.393199 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.393178 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 18:26:07.393310 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.393241 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 18:26:07.393361 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.393319 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jnv68\"" Jul 09 18:26:07.393900 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.393881 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-twfp7" Jul 09 18:26:07.396309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.396284 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 18:26:07.396400 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.396328 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 18:26:07.396458 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.396405 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dqpcm\"" Jul 09 18:26:07.396528 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.396454 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 18:26:07.400983 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.400963 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-twfp7"] Jul 09 18:26:07.404485 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.404462 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4txx9"] Jul 09 18:26:07.544215 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.544177 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a95ca159-d9fe-45bf-85f5-5f6659f03627-cert\") pod \"ingress-canary-twfp7\" (UID: \"a95ca159-d9fe-45bf-85f5-5f6659f03627\") " pod="openshift-ingress-canary/ingress-canary-twfp7" Jul 09 18:26:07.544380 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.544251 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:07.544380 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.544283 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-tmp-dir\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:07.544380 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.544324 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-config-volume\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:07.544380 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.544346 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nglsx\" (UniqueName: \"kubernetes.io/projected/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-kube-api-access-nglsx\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:07.544576 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.544408 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prhrt\" (UniqueName: \"kubernetes.io/projected/a95ca159-d9fe-45bf-85f5-5f6659f03627-kube-api-access-prhrt\") pod \"ingress-canary-twfp7\" (UID: \"a95ca159-d9fe-45bf-85f5-5f6659f03627\") " pod="openshift-ingress-canary/ingress-canary-twfp7" Jul 09 18:26:07.553265 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.553227 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:26:07.553395 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.553230 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:26:07.553452 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.553229 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:26:07.557186 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.557008 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 18:26:07.557807 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.557782 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 18:26:07.558222 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.558015 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 18:26:07.558222 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.558047 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qx56j\"" Jul 09 18:26:07.558367 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.558305 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 18:26:07.558939 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.558921 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7pc9s\"" Jul 09 18:26:07.644976 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.644947 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-config-volume\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:07.645125 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.644981 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nglsx\" (UniqueName: \"kubernetes.io/projected/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-kube-api-access-nglsx\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:07.645125 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.645008 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prhrt\" (UniqueName: \"kubernetes.io/projected/a95ca159-d9fe-45bf-85f5-5f6659f03627-kube-api-access-prhrt\") pod \"ingress-canary-twfp7\" (UID: \"a95ca159-d9fe-45bf-85f5-5f6659f03627\") " pod="openshift-ingress-canary/ingress-canary-twfp7" Jul 09 18:26:07.645125 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.645043 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a95ca159-d9fe-45bf-85f5-5f6659f03627-cert\") pod \"ingress-canary-twfp7\" (UID: \"a95ca159-d9fe-45bf-85f5-5f6659f03627\") " pod="openshift-ingress-canary/ingress-canary-twfp7" Jul 09 18:26:07.645287 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.645144 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:07.645287 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.645192 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-tmp-dir\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:07.645366 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:07.645347 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 18:26:07.645421 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:07.645407 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls podName:d89db1d7-0b3f-4d95-9c61-dd4f92a076ae nodeName:}" failed. No retries permitted until 2026-07-09 18:26:08.145387003 +0000 UTC m=+35.210052662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls") pod "dns-default-4txx9" (UID: "d89db1d7-0b3f-4d95-9c61-dd4f92a076ae") : secret "dns-default-metrics-tls" not found Jul 09 18:26:07.645544 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.645507 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-tmp-dir\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:07.645688 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.645574 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-config-volume\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:07.649481 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.649452 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a95ca159-d9fe-45bf-85f5-5f6659f03627-cert\") pod \"ingress-canary-twfp7\" (UID: \"a95ca159-d9fe-45bf-85f5-5f6659f03627\") " pod="openshift-ingress-canary/ingress-canary-twfp7" Jul 09 18:26:07.662347 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.662317 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nglsx\" (UniqueName: \"kubernetes.io/projected/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-kube-api-access-nglsx\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:07.662813 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.662788 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prhrt\" (UniqueName: \"kubernetes.io/projected/a95ca159-d9fe-45bf-85f5-5f6659f03627-kube-api-access-prhrt\") pod \"ingress-canary-twfp7\" (UID: \"a95ca159-d9fe-45bf-85f5-5f6659f03627\") " pod="openshift-ingress-canary/ingress-canary-twfp7" Jul 09 18:26:07.710645 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.710614 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-twfp7" Jul 09 18:26:07.863076 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:07.863049 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-twfp7"] Jul 09 18:26:07.868050 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:26:07.868018 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95ca159_d9fe_45bf_85f5_5f6659f03627.slice/crio-72ddcf6d83712f482df3716e1f043cabf57a6c55cf362bc5a2ffd0783cc4ab07 WatchSource:0}: Error finding container 72ddcf6d83712f482df3716e1f043cabf57a6c55cf362bc5a2ffd0783cc4ab07: Status 404 returned error can't find the container with id 72ddcf6d83712f482df3716e1f043cabf57a6c55cf362bc5a2ffd0783cc4ab07 Jul 09 18:26:08.149382 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:08.149347 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:08.150045 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:08.149458 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 18:26:08.150045 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:08.149525 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls podName:d89db1d7-0b3f-4d95-9c61-dd4f92a076ae nodeName:}" failed. No retries permitted until 2026-07-09 18:26:09.14950882 +0000 UTC m=+36.214174474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls") pod "dns-default-4txx9" (UID: "d89db1d7-0b3f-4d95-9c61-dd4f92a076ae") : secret "dns-default-metrics-tls" not found Jul 09 18:26:08.694487 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:08.694297 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-twfp7" event={"ID":"a95ca159-d9fe-45bf-85f5-5f6659f03627","Type":"ContainerStarted","Data":"72ddcf6d83712f482df3716e1f043cabf57a6c55cf362bc5a2ffd0783cc4ab07"} Jul 09 18:26:09.155129 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:09.155092 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:09.155551 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:09.155257 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 18:26:09.155551 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:09.155326 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls podName:d89db1d7-0b3f-4d95-9c61-dd4f92a076ae nodeName:}" failed. No retries permitted until 2026-07-09 18:26:11.155308463 +0000 UTC m=+38.219974116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls") pod "dns-default-4txx9" (UID: "d89db1d7-0b3f-4d95-9c61-dd4f92a076ae") : secret "dns-default-metrics-tls" not found Jul 09 18:26:10.133180 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:10.133111 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:10.194433 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:10.194404 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:10.703028 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:10.702949 2558 generic.go:356] "Generic (PLEG): container finished" podID="a93812e1-6915-47c8-b58a-d28334ab03a3" containerID="cb685526ef35d4745241982b21ace8cdb2344d9aab05aec042a6b940bb76bab5" exitCode=0 Jul 09 18:26:10.703028 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:10.703006 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtt72" event={"ID":"a93812e1-6915-47c8-b58a-d28334ab03a3","Type":"ContainerDied","Data":"cb685526ef35d4745241982b21ace8cdb2344d9aab05aec042a6b940bb76bab5"} Jul 09 18:26:11.171686 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:11.171645 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:11.171872 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:11.171819 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 18:26:11.171954 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:11.171906 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls podName:d89db1d7-0b3f-4d95-9c61-dd4f92a076ae nodeName:}" failed. No retries permitted until 2026-07-09 18:26:15.171877869 +0000 UTC m=+42.236543536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls") pod "dns-default-4txx9" (UID: "d89db1d7-0b3f-4d95-9c61-dd4f92a076ae") : secret "dns-default-metrics-tls" not found Jul 09 18:26:11.632978 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:11.632945 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:11.708926 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:11.708887 2558 generic.go:356] "Generic (PLEG): container finished" podID="a93812e1-6915-47c8-b58a-d28334ab03a3" containerID="1c5d3f9f112c322b057c56c35bb42eae911567ae3ca32fc1d9751e65335083b2" exitCode=0 Jul 09 18:26:11.709071 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:11.708946 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtt72" event={"ID":"a93812e1-6915-47c8-b58a-d28334ab03a3","Type":"ContainerDied","Data":"1c5d3f9f112c322b057c56c35bb42eae911567ae3ca32fc1d9751e65335083b2"} Jul 09 18:26:11.725553 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:11.725527 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:11.787500 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:11.787394 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:12.714938 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.714662 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtt72" event={"ID":"a93812e1-6915-47c8-b58a-d28334ab03a3","Type":"ContainerStarted","Data":"5943b2b1d1688a17cccafee99e390adfec491177c4ef42dbb3bee821459cc6e0"} Jul 09 18:26:12.716072 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.716048 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-twfp7" event={"ID":"a95ca159-d9fe-45bf-85f5-5f6659f03627","Type":"ContainerStarted","Data":"c29079c74d9ef7934c2bc35fa02e2e3665d9c296b874bae6f522b39bf98269e5"} Jul 09 18:26:12.765575 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.765532 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qtt72" podStartSLOduration=8.067535044 podStartE2EDuration="39.765518167s" totalImagesPullingTime="31.697983123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:25:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:12.759463317 +0000 UTC m=+39.824128991" watchObservedRunningTime="2026-07-09 18:26:12.765518167 +0000 UTC m=+39.830183840" Jul 09 18:26:12.800580 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.798869 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-twfp7" podStartSLOduration=1.226877181 podStartE2EDuration="5.798852345s" totalImagesPullingTime="4.571975164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:12.795953103 +0000 UTC m=+39.860618777" watchObservedRunningTime="2026-07-09 18:26:12.798852345 +0000 UTC m=+39.863518019" Jul 09 18:26:12.800580 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.799358 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-j9cfj"] Jul 09 18:26:12.812014 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.811991 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:12.814750 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.814559 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 09 18:26:12.814750 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.814642 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 09 18:26:12.815071 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.815047 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-j9cfj"] Jul 09 18:26:12.815419 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.815398 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:26:12.815853 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.815764 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-6xf5t\"" Jul 09 18:26:12.815853 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.815778 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 09 18:26:12.820435 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.820410 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 09 18:26:12.855328 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.855304 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl"] Jul 09 18:26:12.868517 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.868495 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-76c7859458-twqvq"] Jul 09 18:26:12.868633 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.868619 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" Jul 09 18:26:12.875307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.875284 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 09 18:26:12.875559 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.875534 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:26:12.876414 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.876398 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 09 18:26:12.877065 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.877051 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5vvc9\"" Jul 09 18:26:12.877370 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.877352 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:12.882864 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.882842 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 18:26:12.883691 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.883672 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 18:26:12.883794 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.883778 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-j5c65\"" Jul 09 18:26:12.884200 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.884181 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 09 18:26:12.884367 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.884340 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 09 18:26:12.884515 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.884498 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 09 18:26:12.888167 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.888153 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 09 18:26:12.888746 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.888733 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 09 18:26:12.916295 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.916277 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl"] Jul 09 18:26:12.923962 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.923943 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76c7859458-twqvq"] Jul 09 18:26:12.974249 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.974192 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:12.984343 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.984326 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:12.984438 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.984352 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:12.984438 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.984373 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56d0c429-0dc1-4a9f-ae97-c002cc5a74ba-config\") pod \"console-operator-7866497494-j9cfj\" (UID: \"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba\") " pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:12.984543 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.984474 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3b826ae-4a36-452c-a569-1a7cdbb40b06-serving-cert\") pod \"service-ca-operator-7695fc6c8c-v64gl\" (UID: \"a3b826ae-4a36-452c-a569-1a7cdbb40b06\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" Jul 09 18:26:12.984543 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.984508 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-default-certificate\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:12.984619 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.984563 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlbz2\" (UniqueName: \"kubernetes.io/projected/a3b826ae-4a36-452c-a569-1a7cdbb40b06-kube-api-access-hlbz2\") pod \"service-ca-operator-7695fc6c8c-v64gl\" (UID: \"a3b826ae-4a36-452c-a569-1a7cdbb40b06\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" Jul 09 18:26:12.984619 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.984594 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56d0c429-0dc1-4a9f-ae97-c002cc5a74ba-trusted-ca\") pod \"console-operator-7866497494-j9cfj\" (UID: \"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba\") " pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:12.984619 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.984610 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs99q\" (UniqueName: \"kubernetes.io/projected/56d0c429-0dc1-4a9f-ae97-c002cc5a74ba-kube-api-access-vs99q\") pod \"console-operator-7866497494-j9cfj\" (UID: \"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba\") " pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:12.984746 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.984640 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-stats-auth\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:12.984746 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.984657 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56d0c429-0dc1-4a9f-ae97-c002cc5a74ba-serving-cert\") pod \"console-operator-7866497494-j9cfj\" (UID: \"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba\") " pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:12.984746 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.984673 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv7w5\" (UniqueName: \"kubernetes.io/projected/ac02fdbc-8732-4907-8446-81de5cc28330-kube-api-access-lv7w5\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:12.984847 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:12.984754 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3b826ae-4a36-452c-a569-1a7cdbb40b06-config\") pod \"service-ca-operator-7695fc6c8c-v64gl\" (UID: \"a3b826ae-4a36-452c-a569-1a7cdbb40b06\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" Jul 09 18:26:13.050304 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.050270 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:13.085409 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.085368 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3b826ae-4a36-452c-a569-1a7cdbb40b06-config\") pod \"service-ca-operator-7695fc6c8c-v64gl\" (UID: \"a3b826ae-4a36-452c-a569-1a7cdbb40b06\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" Jul 09 18:26:13.085409 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.085413 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:13.085656 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.085431 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:13.085656 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:13.085511 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 18:26:13.085656 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.085536 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56d0c429-0dc1-4a9f-ae97-c002cc5a74ba-config\") pod \"console-operator-7866497494-j9cfj\" (UID: \"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba\") " pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:13.085656 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:13.085549 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle podName:ac02fdbc-8732-4907-8446-81de5cc28330 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:13.58552994 +0000 UTC m=+40.650195617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle") pod "router-default-76c7859458-twqvq" (UID: "ac02fdbc-8732-4907-8446-81de5cc28330") : configmap references non-existent config key: service-ca.crt Jul 09 18:26:13.085656 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:13.085576 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs podName:ac02fdbc-8732-4907-8446-81de5cc28330 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:13.585569147 +0000 UTC m=+40.650234798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs") pod "router-default-76c7859458-twqvq" (UID: "ac02fdbc-8732-4907-8446-81de5cc28330") : secret "router-metrics-certs-default" not found Jul 09 18:26:13.085656 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.085634 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3b826ae-4a36-452c-a569-1a7cdbb40b06-serving-cert\") pod \"service-ca-operator-7695fc6c8c-v64gl\" (UID: \"a3b826ae-4a36-452c-a569-1a7cdbb40b06\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" Jul 09 18:26:13.085656 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.085655 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-default-certificate\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:13.086084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.085811 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlbz2\" (UniqueName: \"kubernetes.io/projected/a3b826ae-4a36-452c-a569-1a7cdbb40b06-kube-api-access-hlbz2\") pod \"service-ca-operator-7695fc6c8c-v64gl\" (UID: \"a3b826ae-4a36-452c-a569-1a7cdbb40b06\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" Jul 09 18:26:13.086084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.085857 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56d0c429-0dc1-4a9f-ae97-c002cc5a74ba-trusted-ca\") pod \"console-operator-7866497494-j9cfj\" (UID: \"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba\") " pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:13.086084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.085882 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs99q\" (UniqueName: \"kubernetes.io/projected/56d0c429-0dc1-4a9f-ae97-c002cc5a74ba-kube-api-access-vs99q\") pod \"console-operator-7866497494-j9cfj\" (UID: \"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba\") " pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:13.086084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.085932 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-stats-auth\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:13.086084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.085958 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56d0c429-0dc1-4a9f-ae97-c002cc5a74ba-serving-cert\") pod \"console-operator-7866497494-j9cfj\" (UID: \"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba\") " pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:13.086084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.085987 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv7w5\" (UniqueName: \"kubernetes.io/projected/ac02fdbc-8732-4907-8446-81de5cc28330-kube-api-access-lv7w5\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:13.086084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.086060 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3b826ae-4a36-452c-a569-1a7cdbb40b06-config\") pod \"service-ca-operator-7695fc6c8c-v64gl\" (UID: \"a3b826ae-4a36-452c-a569-1a7cdbb40b06\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" Jul 09 18:26:13.086390 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.086277 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56d0c429-0dc1-4a9f-ae97-c002cc5a74ba-config\") pod \"console-operator-7866497494-j9cfj\" (UID: \"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba\") " pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:13.087210 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.087188 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56d0c429-0dc1-4a9f-ae97-c002cc5a74ba-trusted-ca\") pod \"console-operator-7866497494-j9cfj\" (UID: \"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba\") " pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:13.090800 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.090771 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3b826ae-4a36-452c-a569-1a7cdbb40b06-serving-cert\") pod \"service-ca-operator-7695fc6c8c-v64gl\" (UID: \"a3b826ae-4a36-452c-a569-1a7cdbb40b06\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" Jul 09 18:26:13.090800 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.090790 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56d0c429-0dc1-4a9f-ae97-c002cc5a74ba-serving-cert\") pod \"console-operator-7866497494-j9cfj\" (UID: \"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba\") " pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:13.090971 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.090904 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-default-certificate\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:13.090971 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.090915 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-stats-auth\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:13.100129 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.100107 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlbz2\" (UniqueName: \"kubernetes.io/projected/a3b826ae-4a36-452c-a569-1a7cdbb40b06-kube-api-access-hlbz2\") pod \"service-ca-operator-7695fc6c8c-v64gl\" (UID: \"a3b826ae-4a36-452c-a569-1a7cdbb40b06\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" Jul 09 18:26:13.100743 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.100689 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs99q\" (UniqueName: \"kubernetes.io/projected/56d0c429-0dc1-4a9f-ae97-c002cc5a74ba-kube-api-access-vs99q\") pod \"console-operator-7866497494-j9cfj\" (UID: \"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba\") " pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:13.100828 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.100772 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv7w5\" (UniqueName: \"kubernetes.io/projected/ac02fdbc-8732-4907-8446-81de5cc28330-kube-api-access-lv7w5\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:13.121033 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.121011 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:13.122953 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.122936 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:13.176988 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.176964 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" Jul 09 18:26:13.217786 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.217759 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:13.267070 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.267043 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-j9cfj"] Jul 09 18:26:13.270008 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:26:13.269977 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d0c429_0dc1_4a9f_ae97_c002cc5a74ba.slice/crio-cff572d7622fbf5369f91d7ffebed147b32454b8b3edabd0aeca7d4e6fdf3180 WatchSource:0}: Error finding container cff572d7622fbf5369f91d7ffebed147b32454b8b3edabd0aeca7d4e6fdf3180: Status 404 returned error can't find the container with id cff572d7622fbf5369f91d7ffebed147b32454b8b3edabd0aeca7d4e6fdf3180 Jul 09 18:26:13.322545 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.322513 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl"] Jul 09 18:26:13.325830 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:26:13.325803 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b826ae_4a36_452c_a569_1a7cdbb40b06.slice/crio-b98b582a7e77e7c15e4cf8393847a458954b352550089ee90b398ff988e207b4 WatchSource:0}: Error finding container b98b582a7e77e7c15e4cf8393847a458954b352550089ee90b398ff988e207b4: Status 404 returned error can't find the container with id b98b582a7e77e7c15e4cf8393847a458954b352550089ee90b398ff988e207b4 Jul 09 18:26:13.589465 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.589385 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:13.589465 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.589419 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:13.589678 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:13.589557 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 18:26:13.589678 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:13.589564 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle podName:ac02fdbc-8732-4907-8446-81de5cc28330 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:14.589545706 +0000 UTC m=+41.654211358 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle") pod "router-default-76c7859458-twqvq" (UID: "ac02fdbc-8732-4907-8446-81de5cc28330") : configmap references non-existent config key: service-ca.crt Jul 09 18:26:13.589678 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:13.589596 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs podName:ac02fdbc-8732-4907-8446-81de5cc28330 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:14.589584966 +0000 UTC m=+41.654250639 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs") pod "router-default-76c7859458-twqvq" (UID: "ac02fdbc-8732-4907-8446-81de5cc28330") : secret "router-metrics-certs-default" not found Jul 09 18:26:13.719118 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.719075 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-j9cfj" event={"ID":"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba","Type":"ContainerStarted","Data":"cff572d7622fbf5369f91d7ffebed147b32454b8b3edabd0aeca7d4e6fdf3180"} Jul 09 18:26:13.720144 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:13.720122 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" event={"ID":"a3b826ae-4a36-452c-a569-1a7cdbb40b06","Type":"ContainerStarted","Data":"b98b582a7e77e7c15e4cf8393847a458954b352550089ee90b398ff988e207b4"} Jul 09 18:26:14.598539 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:14.598501 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:14.598539 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:14.598540 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:14.598767 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:14.598661 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 18:26:14.598767 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:14.598728 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle podName:ac02fdbc-8732-4907-8446-81de5cc28330 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:16.598688472 +0000 UTC m=+43.663354138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle") pod "router-default-76c7859458-twqvq" (UID: "ac02fdbc-8732-4907-8446-81de5cc28330") : configmap references non-existent config key: service-ca.crt Jul 09 18:26:14.598767 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:14.598759 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs podName:ac02fdbc-8732-4907-8446-81de5cc28330 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:16.598748414 +0000 UTC m=+43.663414072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs") pod "router-default-76c7859458-twqvq" (UID: "ac02fdbc-8732-4907-8446-81de5cc28330") : secret "router-metrics-certs-default" not found Jul 09 18:26:15.204110 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:15.204073 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:15.204568 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:15.204229 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 18:26:15.204568 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:15.204291 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls podName:d89db1d7-0b3f-4d95-9c61-dd4f92a076ae nodeName:}" failed. No retries permitted until 2026-07-09 18:26:23.20427491 +0000 UTC m=+50.268940573 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls") pod "dns-default-4txx9" (UID: "d89db1d7-0b3f-4d95-9c61-dd4f92a076ae") : secret "dns-default-metrics-tls" not found Jul 09 18:26:15.712647 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:15.712612 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:15.807819 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:15.807784 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:16.614136 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:16.614100 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:16.614136 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:16.614135 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:16.614516 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:16.614224 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 18:26:16.614516 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:16.614276 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle podName:ac02fdbc-8732-4907-8446-81de5cc28330 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:20.614250492 +0000 UTC m=+47.678916144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle") pod "router-default-76c7859458-twqvq" (UID: "ac02fdbc-8732-4907-8446-81de5cc28330") : configmap references non-existent config key: service-ca.crt Jul 09 18:26:16.614516 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:16.614298 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs podName:ac02fdbc-8732-4907-8446-81de5cc28330 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:20.614291587 +0000 UTC m=+47.678957239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs") pod "router-default-76c7859458-twqvq" (UID: "ac02fdbc-8732-4907-8446-81de5cc28330") : secret "router-metrics-certs-default" not found Jul 09 18:26:16.728714 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:16.728673 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" event={"ID":"a3b826ae-4a36-452c-a569-1a7cdbb40b06","Type":"ContainerStarted","Data":"b1ffefd0e6989aa5ae1fb07e59c62f3230a470cb4cc45c9ebe22dc1a723a8430"} Jul 09 18:26:17.732871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:17.732838 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/0.log" Jul 09 18:26:17.733280 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:17.732890 2558 generic.go:356] "Generic (PLEG): container finished" podID="56d0c429-0dc1-4a9f-ae97-c002cc5a74ba" containerID="2aa096c9a8e6c06125657cc345ef793a87a80deae4331f857b1fed4a9d0814e3" exitCode=255 Jul 09 18:26:17.733280 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:17.732981 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-j9cfj" event={"ID":"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba","Type":"ContainerDied","Data":"2aa096c9a8e6c06125657cc345ef793a87a80deae4331f857b1fed4a9d0814e3"} Jul 09 18:26:17.733280 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:17.733212 2558 scope.go:117] "RemoveContainer" containerID="2aa096c9a8e6c06125657cc345ef793a87a80deae4331f857b1fed4a9d0814e3" Jul 09 18:26:17.788712 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:17.788680 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:17.801108 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:17.798793 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" podStartSLOduration=2.504184552 podStartE2EDuration="5.798774281s" totalImagesPullingTime="3.294589729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:17.798050358 +0000 UTC m=+44.862716032" watchObservedRunningTime="2026-07-09 18:26:17.798774281 +0000 UTC m=+44.863439957" Jul 09 18:26:17.857820 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:17.857789 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:18.001568 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:18.001491 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:18.086893 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:18.086857 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:18.144070 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:18.144038 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:18.243961 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:18.243928 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:18.345152 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:18.345076 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:18.503985 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:18.503958 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:18.594783 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:18.594746 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:18.737072 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:18.737044 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:26:18.737489 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:18.737471 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/0.log" Jul 09 18:26:18.737568 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:18.737512 2558 generic.go:356] "Generic (PLEG): container finished" podID="56d0c429-0dc1-4a9f-ae97-c002cc5a74ba" containerID="49521855fb49dc3b9b1a40c37b4c80c67fd167b82c00f7e0a5ddba3aebb8e14b" exitCode=255 Jul 09 18:26:18.737632 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:18.737564 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-j9cfj" event={"ID":"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba","Type":"ContainerDied","Data":"49521855fb49dc3b9b1a40c37b4c80c67fd167b82c00f7e0a5ddba3aebb8e14b"} Jul 09 18:26:18.737678 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:18.737661 2558 scope.go:117] "RemoveContainer" containerID="2aa096c9a8e6c06125657cc345ef793a87a80deae4331f857b1fed4a9d0814e3" Jul 09 18:26:18.737923 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:18.737898 2558 scope.go:117] "RemoveContainer" containerID="49521855fb49dc3b9b1a40c37b4c80c67fd167b82c00f7e0a5ddba3aebb8e14b" Jul 09 18:26:18.738095 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:18.738078 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-j9cfj_openshift-console-operator(56d0c429-0dc1-4a9f-ae97-c002cc5a74ba)\"" pod="openshift-console-operator/console-operator-7866497494-j9cfj" podUID="56d0c429-0dc1-4a9f-ae97-c002cc5a74ba" Jul 09 18:26:19.066854 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.066773 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zpkd9"] Jul 09 18:26:19.096282 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.096250 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zpkd9"] Jul 09 18:26:19.096413 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.096393 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.099019 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.098990 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 18:26:19.099203 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.099190 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tvx4s\"" Jul 09 18:26:19.099410 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.099398 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 18:26:19.099942 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.099927 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 18:26:19.099989 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.099945 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 18:26:19.105169 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.105145 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:19.135692 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.135666 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.135818 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.135718 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.135818 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.135791 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-crio-socket\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.135915 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.135839 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-data-volume\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.135972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.135906 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vd99\" (UniqueName: \"kubernetes.io/projected/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-api-access-4vd99\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.162170 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.162140 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:19.237152 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.237112 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-data-volume\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.237333 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.237172 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vd99\" (UniqueName: \"kubernetes.io/projected/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-api-access-4vd99\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.237333 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.237229 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.237333 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.237246 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.237333 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.237275 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-crio-socket\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.237563 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:19.237353 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 18:26:19.237563 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:19.237390 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 18:26:19.237563 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:19.237429 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm podName:9f4ee0ac-335c-4580-bae3-16ac3c859dd0 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:19.737411131 +0000 UTC m=+46.802076800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zpkd9" (UID: "9f4ee0ac-335c-4580-bae3-16ac3c859dd0") : configmap "kube-rbac-proxy" not found Jul 09 18:26:19.237563 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:19.237446 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls podName:9f4ee0ac-335c-4580-bae3-16ac3c859dd0 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:19.737438331 +0000 UTC m=+46.802103989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zpkd9" (UID: "9f4ee0ac-335c-4580-bae3-16ac3c859dd0") : secret "insights-runtime-extractor-tls" not found Jul 09 18:26:19.237563 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.237497 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-crio-socket\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.237563 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.237528 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-data-volume\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.251916 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.251889 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vd99\" (UniqueName: \"kubernetes.io/projected/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-api-access-4vd99\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.741430 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.741395 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.741430 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.741428 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:19.741926 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:19.741513 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 18:26:19.741926 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:19.741557 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 18:26:19.741926 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:19.741585 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm podName:9f4ee0ac-335c-4580-bae3-16ac3c859dd0 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:20.741564554 +0000 UTC m=+47.806230219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zpkd9" (UID: "9f4ee0ac-335c-4580-bae3-16ac3c859dd0") : configmap "kube-rbac-proxy" not found Jul 09 18:26:19.741926 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:19.741605 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls podName:9f4ee0ac-335c-4580-bae3-16ac3c859dd0 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:20.741596098 +0000 UTC m=+47.806261750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zpkd9" (UID: "9f4ee0ac-335c-4580-bae3-16ac3c859dd0") : secret "insights-runtime-extractor-tls" not found Jul 09 18:26:19.742083 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.741944 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:26:19.742235 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:19.742223 2558 scope.go:117] "RemoveContainer" containerID="49521855fb49dc3b9b1a40c37b4c80c67fd167b82c00f7e0a5ddba3aebb8e14b" Jul 09 18:26:19.742354 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:19.742338 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-j9cfj_openshift-console-operator(56d0c429-0dc1-4a9f-ae97-c002cc5a74ba)\"" pod="openshift-console-operator/console-operator-7866497494-j9cfj" podUID="56d0c429-0dc1-4a9f-ae97-c002cc5a74ba" Jul 09 18:26:20.104499 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.104428 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-7c88v"] Jul 09 18:26:20.145143 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.145110 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-7c88v"] Jul 09 18:26:20.145290 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.145220 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-7c88v" Jul 09 18:26:20.148004 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.147968 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-kd7dg\"" Jul 09 18:26:20.148144 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.148114 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 09 18:26:20.148210 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.148195 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 09 18:26:20.148988 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.148972 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 09 18:26:20.149111 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.149094 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 09 18:26:20.169424 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.169397 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:20.224063 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.224035 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:20.246364 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.246337 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55-signing-cabundle\") pod \"service-ca-79f68b9f4f-7c88v\" (UID: \"c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7c88v" Jul 09 18:26:20.246471 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.246367 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnhcg\" (UniqueName: \"kubernetes.io/projected/c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55-kube-api-access-hnhcg\") pod \"service-ca-79f68b9f4f-7c88v\" (UID: \"c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7c88v" Jul 09 18:26:20.246557 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.246458 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55-signing-key\") pod \"service-ca-79f68b9f4f-7c88v\" (UID: \"c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7c88v" Jul 09 18:26:20.347996 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.347325 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55-signing-cabundle\") pod \"service-ca-79f68b9f4f-7c88v\" (UID: \"c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7c88v" Jul 09 18:26:20.347996 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.347376 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnhcg\" (UniqueName: \"kubernetes.io/projected/c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55-kube-api-access-hnhcg\") pod \"service-ca-79f68b9f4f-7c88v\" (UID: \"c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7c88v" Jul 09 18:26:20.347996 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.347462 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55-signing-key\") pod \"service-ca-79f68b9f4f-7c88v\" (UID: \"c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7c88v" Jul 09 18:26:20.350967 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.350936 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55-signing-key\") pod \"service-ca-79f68b9f4f-7c88v\" (UID: \"c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7c88v" Jul 09 18:26:20.351110 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.351090 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55-signing-cabundle\") pod \"service-ca-79f68b9f4f-7c88v\" (UID: \"c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7c88v" Jul 09 18:26:20.359320 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.359267 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnhcg\" (UniqueName: \"kubernetes.io/projected/c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55-kube-api-access-hnhcg\") pod \"service-ca-79f68b9f4f-7c88v\" (UID: \"c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7c88v" Jul 09 18:26:20.453719 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.453668 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-7c88v" Jul 09 18:26:20.595198 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.595166 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-7c88v"] Jul 09 18:26:20.598098 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:26:20.598071 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d05e59_3ad2_4f9c_bb68_7a64e1ab2c55.slice/crio-cf3e28be2efe6470d07337dc746de01daac4e2b0d6850f71103bbdc9eb004195 WatchSource:0}: Error finding container cf3e28be2efe6470d07337dc746de01daac4e2b0d6850f71103bbdc9eb004195: Status 404 returned error can't find the container with id cf3e28be2efe6470d07337dc746de01daac4e2b0d6850f71103bbdc9eb004195 Jul 09 18:26:20.649865 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.649844 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:20.649956 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.649872 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:20.650057 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:20.649979 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 18:26:20.650057 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:20.650015 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle podName:ac02fdbc-8732-4907-8446-81de5cc28330 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:28.649996077 +0000 UTC m=+55.714661749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle") pod "router-default-76c7859458-twqvq" (UID: "ac02fdbc-8732-4907-8446-81de5cc28330") : configmap references non-existent config key: service-ca.crt Jul 09 18:26:20.650057 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:20.650037 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs podName:ac02fdbc-8732-4907-8446-81de5cc28330 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:28.650029123 +0000 UTC m=+55.714694774 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs") pod "router-default-76c7859458-twqvq" (UID: "ac02fdbc-8732-4907-8446-81de5cc28330") : secret "router-metrics-certs-default" not found Jul 09 18:26:20.746456 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.746426 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-7c88v" event={"ID":"c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55","Type":"ContainerStarted","Data":"42ab6b9baa80c6384784e051bbdb7a43ed7349c8d8327a20952f85c54e2390ed"} Jul 09 18:26:20.746813 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.746471 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-7c88v" event={"ID":"c4d05e59-3ad2-4f9c-bb68-7a64e1ab2c55","Type":"ContainerStarted","Data":"cf3e28be2efe6470d07337dc746de01daac4e2b0d6850f71103bbdc9eb004195"} Jul 09 18:26:20.751050 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.751033 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:20.751121 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.751058 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:20.751157 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:20.751140 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 18:26:20.751197 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:20.751155 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 18:26:20.751197 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:20.751194 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm podName:9f4ee0ac-335c-4580-bae3-16ac3c859dd0 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:22.751180009 +0000 UTC m=+49.815845661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zpkd9" (UID: "9f4ee0ac-335c-4580-bae3-16ac3c859dd0") : configmap "kube-rbac-proxy" not found Jul 09 18:26:20.751272 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:20.751207 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls podName:9f4ee0ac-335c-4580-bae3-16ac3c859dd0 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:22.751201066 +0000 UTC m=+49.815866717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zpkd9" (UID: "9f4ee0ac-335c-4580-bae3-16ac3c859dd0") : secret "insights-runtime-extractor-tls" not found Jul 09 18:26:20.775224 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:20.775182 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-7c88v" podStartSLOduration=0.775167488 podStartE2EDuration="775.167488ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:20.774240242 +0000 UTC m=+47.838905918" watchObservedRunningTime="2026-07-09 18:26:20.775167488 +0000 UTC m=+47.839833161" Jul 09 18:26:21.045946 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:21.045859 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:21.948605 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:21.948573 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:22.088236 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:22.088204 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tvjd5_47cb0898-a067-4eb1-afe9-e2b5413b2cee/dns-node-resolver/0.log" Jul 09 18:26:22.219048 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:22.218939 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:22.315996 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:22.315954 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:22.436237 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:22.436192 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:22.518998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:22.518909 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:22.632431 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:22.632398 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:22.677472 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:22.677447 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4b29x_03116b04-5653-4707-8b77-a6c937981f49/node-ca/0.log" Jul 09 18:26:22.767867 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:22.767833 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:22.767867 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:22.767866 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:22.768065 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:22.767955 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 18:26:22.768065 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:22.767997 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 18:26:22.768065 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:22.768020 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm podName:9f4ee0ac-335c-4580-bae3-16ac3c859dd0 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:26.768000803 +0000 UTC m=+53.832666499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zpkd9" (UID: "9f4ee0ac-335c-4580-bae3-16ac3c859dd0") : configmap "kube-rbac-proxy" not found Jul 09 18:26:22.768065 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:22.768038 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls podName:9f4ee0ac-335c-4580-bae3-16ac3c859dd0 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:26.768028659 +0000 UTC m=+53.832694317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zpkd9" (UID: "9f4ee0ac-335c-4580-bae3-16ac3c859dd0") : secret "insights-runtime-extractor-tls" not found Jul 09 18:26:23.123328 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:23.123294 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:23.123669 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:23.123349 2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:23.123761 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:23.123746 2558 scope.go:117] "RemoveContainer" containerID="49521855fb49dc3b9b1a40c37b4c80c67fd167b82c00f7e0a5ddba3aebb8e14b" Jul 09 18:26:23.123917 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:23.123895 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-j9cfj_openshift-console-operator(56d0c429-0dc1-4a9f-ae97-c002cc5a74ba)\"" pod="openshift-console-operator/console-operator-7866497494-j9cfj" podUID="56d0c429-0dc1-4a9f-ae97-c002cc5a74ba" Jul 09 18:26:23.272041 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:23.272011 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:23.272235 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:23.272133 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 18:26:23.272235 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:23.272201 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls podName:d89db1d7-0b3f-4d95-9c61-dd4f92a076ae nodeName:}" failed. No retries permitted until 2026-07-09 18:26:39.27218306 +0000 UTC m=+66.336848713 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls") pod "dns-default-4txx9" (UID: "d89db1d7-0b3f-4d95-9c61-dd4f92a076ae") : secret "dns-default-metrics-tls" not found Jul 09 18:26:23.635159 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:23.635118 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:23.678806 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:23.678771 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-twfp7_a95ca159-d9fe-45bf-85f5-5f6659f03627/serve-healthcheck-canary/0.log" Jul 09 18:26:24.024147 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:24.024070 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:24.094682 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:24.094650 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:24.231892 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:24.231860 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:24.422954 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:24.422897 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:24.629393 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:24.629364 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:25.432794 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:25.432755 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:25.624729 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:25.624685 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:25.934810 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:25.934778 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:26.035736 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:26.035687 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:26.228904 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:26.228811 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:26.422242 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:26.422204 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:26.800672 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:26.800638 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:26.800672 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:26.800683 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:26.801180 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:26.800775 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 18:26:26.801180 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:26.800843 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm podName:9f4ee0ac-335c-4580-bae3-16ac3c859dd0 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:34.80082771 +0000 UTC m=+61.865493365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zpkd9" (UID: "9f4ee0ac-335c-4580-bae3-16ac3c859dd0") : configmap "kube-rbac-proxy" not found Jul 09 18:26:26.801180 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:26.800895 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 18:26:26.801180 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:26.800957 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls podName:9f4ee0ac-335c-4580-bae3-16ac3c859dd0 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:34.800939952 +0000 UTC m=+61.865605611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zpkd9" (UID: "9f4ee0ac-335c-4580-bae3-16ac3c859dd0") : secret "insights-runtime-extractor-tls" not found Jul 09 18:26:26.830747 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:26.830692 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:26.880156 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:26.880127 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:27.231879 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:27.231844 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:27.491002 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:27.490912 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:28.032354 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:28.032317 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:28.236728 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:28.236685 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:28.635843 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:28.635811 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:28.715646 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:28.715605 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:28.715646 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:28.715643 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:28.716225 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:28.716204 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac02fdbc-8732-4907-8446-81de5cc28330-service-ca-bundle\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:28.717983 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:28.717962 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac02fdbc-8732-4907-8446-81de5cc28330-metrics-certs\") pod \"router-default-76c7859458-twqvq\" (UID: \"ac02fdbc-8732-4907-8446-81de5cc28330\") " pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:28.785758 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:28.785719 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:28.832387 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:28.832282 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:28.914003 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:28.913913 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76c7859458-twqvq"] Jul 09 18:26:28.917208 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:26:28.917180 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac02fdbc_8732_4907_8446_81de5cc28330.slice/crio-83d4b385e3bc11f847295ffbaec49ca143e1e1a4d136d99cac88cbf2d41bf92f WatchSource:0}: Error finding container 83d4b385e3bc11f847295ffbaec49ca143e1e1a4d136d99cac88cbf2d41bf92f: Status 404 returned error can't find the container with id 83d4b385e3bc11f847295ffbaec49ca143e1e1a4d136d99cac88cbf2d41bf92f Jul 09 18:26:29.137004 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:29.136974 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:29.237414 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:29.237335 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:29.770518 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:29.770479 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76c7859458-twqvq" event={"ID":"ac02fdbc-8732-4907-8446-81de5cc28330","Type":"ContainerStarted","Data":"30be47cf2114e69142ead7e5848024d012e71e11ab4a72dc9a6295792fdc61db"} Jul 09 18:26:29.770518 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:29.770526 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76c7859458-twqvq" event={"ID":"ac02fdbc-8732-4907-8446-81de5cc28330","Type":"ContainerStarted","Data":"83d4b385e3bc11f847295ffbaec49ca143e1e1a4d136d99cac88cbf2d41bf92f"} Jul 09 18:26:29.786862 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:29.786837 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:29.789446 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:29.789422 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:29.794167 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:29.794128 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-76c7859458-twqvq" podStartSLOduration=17.794115568 podStartE2EDuration="17.794115568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:29.793836737 +0000 UTC m=+56.858502411" watchObservedRunningTime="2026-07-09 18:26:29.794115568 +0000 UTC m=+56.858781247" Jul 09 18:26:30.029339 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:30.029260 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:30.427740 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:30.427690 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:30.624935 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:30.624901 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:30.773437 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:30.773356 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:30.774636 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:30.774615 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76c7859458-twqvq" Jul 09 18:26:30.830879 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:30.830847 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:31.112203 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:31.112123 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:31.225985 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:31.225950 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:31.841492 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:31.841457 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:32.028237 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:32.028193 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:32.428275 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:32.428239 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:32.638641 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:32.638611 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:32.899819 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:32.899786 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:33.696401 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:33.696372 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmxck" Jul 09 18:26:34.860981 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:34.860949 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:34.860981 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:34.860987 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:34.861586 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:34.861564 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:34.863273 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:34.863255 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4ee0ac-335c-4580-bae3-16ac3c859dd0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpkd9\" (UID: \"9f4ee0ac-335c-4580-bae3-16ac3c859dd0\") " pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:35.008868 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:35.008841 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tvx4s\"" Jul 09 18:26:35.016992 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:35.016974 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zpkd9" Jul 09 18:26:35.166228 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:35.166195 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zpkd9"] Jul 09 18:26:35.169073 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:26:35.169049 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4ee0ac_335c_4580_bae3_16ac3c859dd0.slice/crio-232b8e3fc0684a5a4cd74ed888929404c650140a667b523f3a0eca375c96a5eb WatchSource:0}: Error finding container 232b8e3fc0684a5a4cd74ed888929404c650140a667b523f3a0eca375c96a5eb: Status 404 returned error can't find the container with id 232b8e3fc0684a5a4cd74ed888929404c650140a667b523f3a0eca375c96a5eb Jul 09 18:26:35.788592 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:35.788542 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zpkd9" event={"ID":"9f4ee0ac-335c-4580-bae3-16ac3c859dd0","Type":"ContainerStarted","Data":"232b8e3fc0684a5a4cd74ed888929404c650140a667b523f3a0eca375c96a5eb"} Jul 09 18:26:37.554247 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:37.554220 2558 scope.go:117] "RemoveContainer" containerID="49521855fb49dc3b9b1a40c37b4c80c67fd167b82c00f7e0a5ddba3aebb8e14b" Jul 09 18:26:38.800121 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:38.800104 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:26:38.800425 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:38.800169 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-j9cfj" event={"ID":"56d0c429-0dc1-4a9f-ae97-c002cc5a74ba","Type":"ContainerStarted","Data":"580d0d08646a1331e7b945e8191127c1eb48cab747473ed92ed4ad98253513f4"} Jul 09 18:26:38.800485 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:38.800439 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:38.801886 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:38.801848 2558 patch_prober.go:28] interesting pod/console-operator-7866497494-j9cfj container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.134.0.8:8443/readyz\": dial tcp 10.134.0.8:8443: connect: connection refused" start-of-body= Jul 09 18:26:38.801999 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:38.801918 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7866497494-j9cfj" podUID="56d0c429-0dc1-4a9f-ae97-c002cc5a74ba" containerName="console-operator" probeResult="failure" output="Get \"https://10.134.0.8:8443/readyz\": dial tcp 10.134.0.8:8443: connect: connection refused" Jul 09 18:26:38.818115 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:38.818076 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-j9cfj" podStartSLOduration=23.46344693 podStartE2EDuration="26.818061506s" totalImagesPullingTime="3.354614576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:38.817069627 +0000 UTC m=+65.881735302" watchObservedRunningTime="2026-07-09 18:26:38.818061506 +0000 UTC m=+65.882727180" Jul 09 18:26:39.092912 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.092804 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fxn\" (UniqueName: \"kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn\") pod \"network-check-target-j2xsk\" (UID: \"0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e\") " pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:26:39.092912 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.092865 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs\") pod \"network-metrics-daemon-r5jrm\" (UID: \"e673ec85-406b-4221-8e0a-441d1886d20d\") " pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:26:39.095564 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.095544 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 18:26:39.095635 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.095601 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 18:26:39.105926 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.105900 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e673ec85-406b-4221-8e0a-441d1886d20d-metrics-certs\") pod \"network-metrics-daemon-r5jrm\" (UID: \"e673ec85-406b-4221-8e0a-441d1886d20d\") " pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:26:39.106120 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.106105 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 18:26:39.116416 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.116395 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6fxn\" (UniqueName: \"kubernetes.io/projected/0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e-kube-api-access-p6fxn\") pod \"network-check-target-j2xsk\" (UID: \"0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e\") " pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:26:39.193536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.193496 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:26:39.196391 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.196372 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 18:26:39.206411 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.206390 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cfe536a-207f-4ce7-8e2a-6a68d85fc813-original-pull-secret\") pod \"global-pull-secret-syncer-r7g5z\" (UID: \"8cfe536a-207f-4ce7-8e2a-6a68d85fc813\") " pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:26:39.294082 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.294052 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:39.296835 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.296809 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d89db1d7-0b3f-4d95-9c61-dd4f92a076ae-metrics-tls\") pod \"dns-default-4txx9\" (UID: \"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae\") " pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:39.365091 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.365003 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7g5z" Jul 09 18:26:39.374431 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.374413 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qx56j\"" Jul 09 18:26:39.379972 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.379951 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7pc9s\"" Jul 09 18:26:39.381949 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.381924 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:26:39.387846 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.387821 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5jrm" Jul 09 18:26:39.427018 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.424276 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:39.483163 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.482518 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:39.510344 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.505787 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jnv68\"" Jul 09 18:26:39.514007 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.513895 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:39.540828 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.540356 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r7g5z"] Jul 09 18:26:39.563721 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.562890 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j2xsk"] Jul 09 18:26:39.580581 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.580145 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r5jrm"] Jul 09 18:26:39.658540 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:26:39.658509 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode673ec85_406b_4221_8e0a_441d1886d20d.slice/crio-e81a414ab3b8378a7a43e781bbc35ee3b7eea94646b4c0803c3943d3685a5f12 WatchSource:0}: Error finding container e81a414ab3b8378a7a43e781bbc35ee3b7eea94646b4c0803c3943d3685a5f12: Status 404 returned error can't find the container with id e81a414ab3b8378a7a43e781bbc35ee3b7eea94646b4c0803c3943d3685a5f12 Jul 09 18:26:39.658830 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:26:39.658812 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bec4d50_3deb_403b_9ba6_a09e4ebe3a0e.slice/crio-946d332471855c6031c0c229fa56c0f3eb36e2b7db806fd868f7feffe7046f8b WatchSource:0}: Error finding container 946d332471855c6031c0c229fa56c0f3eb36e2b7db806fd868f7feffe7046f8b: Status 404 returned error can't find the container with id 946d332471855c6031c0c229fa56c0f3eb36e2b7db806fd868f7feffe7046f8b Jul 09 18:26:39.666770 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.666149 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4txx9"] Jul 09 18:26:39.672435 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:26:39.672415 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89db1d7_0b3f_4d95_9c61_dd4f92a076ae.slice/crio-6927e19ee49b27b88fe5b94615205721b42215025b5f24ef34e79334f2fcc6d5 WatchSource:0}: Error finding container 6927e19ee49b27b88fe5b94615205721b42215025b5f24ef34e79334f2fcc6d5: Status 404 returned error can't find the container with id 6927e19ee49b27b88fe5b94615205721b42215025b5f24ef34e79334f2fcc6d5 Jul 09 18:26:39.805055 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.805028 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4txx9" event={"ID":"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae","Type":"ContainerStarted","Data":"6927e19ee49b27b88fe5b94615205721b42215025b5f24ef34e79334f2fcc6d5"} Jul 09 18:26:39.806244 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.806219 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r7g5z" event={"ID":"8cfe536a-207f-4ce7-8e2a-6a68d85fc813","Type":"ContainerStarted","Data":"17ffbb65b93a6b2e4daa6954d60175c32262ffa4f0f90db6751bcd21d3f0c114"} Jul 09 18:26:39.808057 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.808036 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zpkd9" event={"ID":"9f4ee0ac-335c-4580-bae3-16ac3c859dd0","Type":"ContainerStarted","Data":"5279776257d337726638c1481d4f284543389897e361dba5868a85eee8a97ba7"} Jul 09 18:26:39.808142 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.808082 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zpkd9" event={"ID":"9f4ee0ac-335c-4580-bae3-16ac3c859dd0","Type":"ContainerStarted","Data":"008a2693151e41fa43cd1092a0615fed46abf9723e9c33149e38bfddf127bb51"} Jul 09 18:26:39.809042 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.809015 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5jrm" event={"ID":"e673ec85-406b-4221-8e0a-441d1886d20d","Type":"ContainerStarted","Data":"e81a414ab3b8378a7a43e781bbc35ee3b7eea94646b4c0803c3943d3685a5f12"} Jul 09 18:26:39.810051 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.810034 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j2xsk" event={"ID":"0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e","Type":"ContainerStarted","Data":"946d332471855c6031c0c229fa56c0f3eb36e2b7db806fd868f7feffe7046f8b"} Jul 09 18:26:39.931689 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.931508 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm"] Jul 09 18:26:39.958477 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.958443 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-27t6j"] Jul 09 18:26:39.958649 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.958582 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:39.963593 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.963570 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 09 18:26:39.963863 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.963824 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 09 18:26:39.963993 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.963901 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 09 18:26:39.963993 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.963900 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 09 18:26:39.964373 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.964192 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 09 18:26:39.964802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.964776 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 09 18:26:39.965120 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.965100 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 09 18:26:39.974484 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.974463 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-27t6j"] Jul 09 18:26:39.974581 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.974494 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm"] Jul 09 18:26:39.974581 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.974567 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-27t6j" Jul 09 18:26:39.977575 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.977558 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 09 18:26:39.978149 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.977578 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9t5fz\"" Jul 09 18:26:39.978149 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:39.977889 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 09 18:26:40.000404 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.000382 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a989340-fe00-4b39-ad17-58633d4bd584-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.000521 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.000416 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8lgg\" (UniqueName: \"kubernetes.io/projected/2a989340-fe00-4b39-ad17-58633d4bd584-kube-api-access-p8lgg\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.000521 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.000437 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a989340-fe00-4b39-ad17-58633d4bd584-ca\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.000638 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.000522 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a989340-fe00-4b39-ad17-58633d4bd584-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.000638 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.000568 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a989340-fe00-4b39-ad17-58633d4bd584-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.000791 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.000651 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a989340-fe00-4b39-ad17-58633d4bd584-hub\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.031634 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.031604 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:40.057925 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.057904 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-j9cfj" Jul 09 18:26:40.101122 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.101086 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/995ece05-95be-45f0-acb3-1469eb38f416-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-27t6j\" (UID: \"995ece05-95be-45f0-acb3-1469eb38f416\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-27t6j" Jul 09 18:26:40.101294 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.101136 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a989340-fe00-4b39-ad17-58633d4bd584-hub\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.101294 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.101172 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/995ece05-95be-45f0-acb3-1469eb38f416-nginx-conf\") pod \"networking-console-plugin-6cf856758c-27t6j\" (UID: \"995ece05-95be-45f0-acb3-1469eb38f416\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-27t6j" Jul 09 18:26:40.101294 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.101205 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a989340-fe00-4b39-ad17-58633d4bd584-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.101294 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.101243 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8lgg\" (UniqueName: \"kubernetes.io/projected/2a989340-fe00-4b39-ad17-58633d4bd584-kube-api-access-p8lgg\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.101294 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.101275 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a989340-fe00-4b39-ad17-58633d4bd584-ca\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.101511 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.101297 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a989340-fe00-4b39-ad17-58633d4bd584-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.101511 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.101323 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a989340-fe00-4b39-ad17-58633d4bd584-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.102665 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.102639 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a989340-fe00-4b39-ad17-58633d4bd584-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.104629 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.104602 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a989340-fe00-4b39-ad17-58633d4bd584-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.104789 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.104747 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a989340-fe00-4b39-ad17-58633d4bd584-hub\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.105668 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.105611 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a989340-fe00-4b39-ad17-58633d4bd584-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.107749 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.107694 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a989340-fe00-4b39-ad17-58633d4bd584-ca\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.115604 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.115581 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8lgg\" (UniqueName: \"kubernetes.io/projected/2a989340-fe00-4b39-ad17-58633d4bd584-kube-api-access-p8lgg\") pod \"cluster-proxy-proxy-agent-8d5557994-hcmhm\" (UID: \"2a989340-fe00-4b39-ad17-58633d4bd584\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.123508 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.123480 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:40.202233 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.202146 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/995ece05-95be-45f0-acb3-1469eb38f416-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-27t6j\" (UID: \"995ece05-95be-45f0-acb3-1469eb38f416\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-27t6j" Jul 09 18:26:40.202233 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.202200 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/995ece05-95be-45f0-acb3-1469eb38f416-nginx-conf\") pod \"networking-console-plugin-6cf856758c-27t6j\" (UID: \"995ece05-95be-45f0-acb3-1469eb38f416\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-27t6j" Jul 09 18:26:40.203103 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.203074 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/995ece05-95be-45f0-acb3-1469eb38f416-nginx-conf\") pod \"networking-console-plugin-6cf856758c-27t6j\" (UID: \"995ece05-95be-45f0-acb3-1469eb38f416\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-27t6j" Jul 09 18:26:40.205052 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.205016 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/995ece05-95be-45f0-acb3-1469eb38f416-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-27t6j\" (UID: \"995ece05-95be-45f0-acb3-1469eb38f416\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-27t6j" Jul 09 18:26:40.228508 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.228455 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:40.277212 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.276933 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" Jul 09 18:26:40.285467 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.285419 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-27t6j" Jul 09 18:26:40.325537 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.325500 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:40.435417 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.435123 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:40.469575 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.469408 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm"] Jul 09 18:26:40.474379 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:26:40.474341 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a989340_fe00_4b39_ad17_58633d4bd584.slice/crio-13828c7340f85c87cfdd078adf49ef311eddf78e937fdfd1b0ee232a9ee1fb3a WatchSource:0}: Error finding container 13828c7340f85c87cfdd078adf49ef311eddf78e937fdfd1b0ee232a9ee1fb3a: Status 404 returned error can't find the container with id 13828c7340f85c87cfdd078adf49ef311eddf78e937fdfd1b0ee232a9ee1fb3a Jul 09 18:26:40.501431 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.501359 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-27t6j"] Jul 09 18:26:40.508797 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:26:40.508765 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995ece05_95be_45f0_acb3_1469eb38f416.slice/crio-25c3928ea9a36edb6c98c2673d78d75effcb7fcc5c1eb5e4ee39383168731cd0 WatchSource:0}: Error finding container 25c3928ea9a36edb6c98c2673d78d75effcb7fcc5c1eb5e4ee39383168731cd0: Status 404 returned error can't find the container with id 25c3928ea9a36edb6c98c2673d78d75effcb7fcc5c1eb5e4ee39383168731cd0 Jul 09 18:26:40.545203 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.545171 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:40.717318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.717285 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:40.799746 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.799598 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:40.815539 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.815506 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-27t6j" event={"ID":"995ece05-95be-45f0-acb3-1469eb38f416","Type":"ContainerStarted","Data":"25c3928ea9a36edb6c98c2673d78d75effcb7fcc5c1eb5e4ee39383168731cd0"} Jul 09 18:26:40.817727 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:40.817655 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" event={"ID":"2a989340-fe00-4b39-ad17-58633d4bd584","Type":"ContainerStarted","Data":"13828c7340f85c87cfdd078adf49ef311eddf78e937fdfd1b0ee232a9ee1fb3a"} Jul 09 18:26:41.116712 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:41.116601 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:41.174166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:41.174132 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:41.315238 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:41.315103 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:41.372542 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:41.372458 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:41.728568 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:41.728535 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:41.790651 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:41.790561 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:41.825983 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:41.825882 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5jrm" event={"ID":"e673ec85-406b-4221-8e0a-441d1886d20d","Type":"ContainerStarted","Data":"0143f30f3e8a303c75e5585363f4b7cceb2f3293357f994571d6871358d27db8"} Jul 09 18:26:41.959236 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:41.959204 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:42.061756 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:42.061716 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:42.970962 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:42.970842 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:48.026692 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:48.026600 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:48.266497 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:48.266451 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:48.769774 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:48.769737 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:49.858986 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.858945 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4txx9" event={"ID":"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae","Type":"ContainerStarted","Data":"becab5152928e92257a14b524b2a5b481e4d3ea12b07f6507921bea0b3613b1b"} Jul 09 18:26:49.859423 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.859004 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4txx9" event={"ID":"d89db1d7-0b3f-4d95-9c61-dd4f92a076ae","Type":"ContainerStarted","Data":"ebc4605cd916cd3f666f0d3e4460ead89bd0b821dab1151b86a1a12fb23e68a3"} Jul 09 18:26:49.859423 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.859028 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:49.860323 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.860303 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r7g5z" event={"ID":"8cfe536a-207f-4ce7-8e2a-6a68d85fc813","Type":"ContainerStarted","Data":"66bd338b4a9b00ebfa650808899e666d00bfe757d2871d20af901cb04b79791c"} Jul 09 18:26:49.862190 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.862168 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zpkd9" event={"ID":"9f4ee0ac-335c-4580-bae3-16ac3c859dd0","Type":"ContainerStarted","Data":"0c47760595682f6f8b76fd7d4d37a6916dd7b18026a21764c9bdd67caefeec82"} Jul 09 18:26:49.863743 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.863718 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5jrm" event={"ID":"e673ec85-406b-4221-8e0a-441d1886d20d","Type":"ContainerStarted","Data":"67f93a13712a3054cab9f5e33cf64596a8e266fdfee2e043e29a18d60426884a"} Jul 09 18:26:49.864947 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.864929 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j2xsk" event={"ID":"0bec4d50-3deb-403b-9ba6-a09e4ebe3a0e","Type":"ContainerStarted","Data":"ee14a3242b759b7b91c7d6e21cde6152dba9ac8a49df12222e3d8d5264eef8a9"} Jul 09 18:26:49.865014 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.864964 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:26:49.866240 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.866223 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-27t6j" event={"ID":"995ece05-95be-45f0-acb3-1469eb38f416","Type":"ContainerStarted","Data":"cc01272dbc5588f2c25ae93c616964b1361edaa299f1ae11238e9566cad78b20"} Jul 09 18:26:49.871640 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.871619 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" event={"ID":"2a989340-fe00-4b39-ad17-58633d4bd584","Type":"ContainerStarted","Data":"5cf85b1de44a9004e2bf6b13bb80652ca7cbf3df37da717a2be14c3ba6ca4d37"} Jul 09 18:26:49.871741 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.871648 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" event={"ID":"2a989340-fe00-4b39-ad17-58633d4bd584","Type":"ContainerStarted","Data":"d1e01bc7ebd888a693ddfccfdb6a53b7569b5c47e8041cfddfc7bb4e1485b40a"} Jul 09 18:26:49.871741 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.871660 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" event={"ID":"2a989340-fe00-4b39-ad17-58633d4bd584","Type":"ContainerStarted","Data":"f52c85bd50a3158ec50e1c79ff7274cc5ab1825aca4b279a4be0502df4c16d0a"} Jul 09 18:26:49.880171 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.880134 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4txx9" podStartSLOduration=33.469514428 podStartE2EDuration="42.880123307s" totalImagesPullingTime="9.410608879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:49.878618387 +0000 UTC m=+76.943284061" watchObservedRunningTime="2026-07-09 18:26:49.880123307 +0000 UTC m=+76.944788981" Jul 09 18:26:49.898139 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.898097 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r5jrm" podStartSLOduration=75.024372099 podStartE2EDuration="1m16.898084621s" totalImagesPullingTime="1.873712522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:49.896721835 +0000 UTC m=+76.961387500" watchObservedRunningTime="2026-07-09 18:26:49.898084621 +0000 UTC m=+76.962750289" Jul 09 18:26:49.912793 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.912744 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j2xsk" podStartSLOduration=67.443947016 podStartE2EDuration="1m16.912727201s" totalImagesPullingTime="9.468780185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:49.911680543 +0000 UTC m=+76.976346210" watchObservedRunningTime="2026-07-09 18:26:49.912727201 +0000 UTC m=+76.977392879" Jul 09 18:26:49.931496 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.931425 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8d5557994-hcmhm" podStartSLOduration=2.255599464 podStartE2EDuration="10.931411344s" totalImagesPullingTime="8.67581188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:49.930354315 +0000 UTC m=+76.995019988" watchObservedRunningTime="2026-07-09 18:26:49.931411344 +0000 UTC m=+76.996077017" Jul 09 18:26:49.952319 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.952281 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zpkd9" podStartSLOduration=17.185925396 podStartE2EDuration="30.952269537s" totalImagesPullingTime="13.766344141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:26:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:49.951220401 +0000 UTC m=+77.015886075" watchObservedRunningTime="2026-07-09 18:26:49.952269537 +0000 UTC m=+77.016935210" Jul 09 18:26:49.967019 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.966981 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-27t6j" podStartSLOduration=2.393650857 podStartE2EDuration="10.966966167s" totalImagesPullingTime="8.57331531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:49.966576249 +0000 UTC m=+77.031241924" watchObservedRunningTime="2026-07-09 18:26:49.966966167 +0000 UTC m=+77.031631842" Jul 09 18:26:49.985050 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:49.984984 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-r7g5z" podStartSLOduration=66.404972271 podStartE2EDuration="1m15.984972851s" totalImagesPullingTime="9.58000058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:49.984357545 +0000 UTC m=+77.049023219" watchObservedRunningTime="2026-07-09 18:26:49.984972851 +0000 UTC m=+77.049638556" Jul 09 18:26:50.013609 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:50.013576 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:50.075904 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:50.075880 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:50.129389 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:50.129331 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:50.193356 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:50.193326 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:50.368558 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:50.368527 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:50.425368 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:50.425340 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:53.287172 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:53.287142 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:53.350625 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:53.350592 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:53.741242 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:53.741197 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:54.425084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:54.425054 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:54.495968 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:54.495941 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:54.679899 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:54.679820 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:54.730298 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:54.730265 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:54.952882 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:54.952811 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:55.018118 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:55.018088 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:55.544364 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:55.544332 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:55.645927 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:55.645896 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:56.146568 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:56.146538 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:58.471729 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.471684 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:58.567865 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.567828 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:58.737756 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.737668 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:58.835097 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.835061 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8m9g4"] Jul 09 18:26:58.835294 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.835124 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:58.839841 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.839820 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:58.842612 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.842589 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 18:26:58.842744 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.842686 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 18:26:58.842833 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.842814 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 18:26:58.843604 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.843557 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 18:26:58.843604 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.843567 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 18:26:58.843604 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.843584 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xcq5v\"" Jul 09 18:26:58.844048 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.844025 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 18:26:58.908599 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.908567 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:58.955744 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.955716 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:58.955904 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.955756 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e74848c0-7cd6-4754-936c-e4733c3764f8-sys\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:58.955904 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.955774 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-tls\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:58.955904 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.955855 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-wtmp\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:58.955904 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.955880 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e74848c0-7cd6-4754-936c-e4733c3764f8-metrics-client-ca\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:58.955904 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.955899 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd4cv\" (UniqueName: \"kubernetes.io/projected/e74848c0-7cd6-4754-936c-e4733c3764f8-kube-api-access-kd4cv\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:58.956084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.955918 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-textfile\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:58.956084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.955944 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:58.956084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:58.955971 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e74848c0-7cd6-4754-936c-e4733c3764f8-root\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.015579 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.015501 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:59.057290 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.057255 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-wtmp\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.057290 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.057294 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e74848c0-7cd6-4754-936c-e4733c3764f8-metrics-client-ca\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.057512 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.057314 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd4cv\" (UniqueName: \"kubernetes.io/projected/e74848c0-7cd6-4754-936c-e4733c3764f8-kube-api-access-kd4cv\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.057512 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.057332 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-textfile\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.057512 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.057360 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.057512 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.057411 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e74848c0-7cd6-4754-936c-e4733c3764f8-root\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.057512 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.057449 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.057512 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.057464 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-wtmp\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.057512 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.057488 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e74848c0-7cd6-4754-936c-e4733c3764f8-sys\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.057512 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.057509 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-tls\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.057923 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:59.057619 2558 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 09 18:26:59.057923 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:26:59.057674 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-tls podName:e74848c0-7cd6-4754-936c-e4733c3764f8 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:59.557655101 +0000 UTC m=+86.622320752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-tls") pod "node-exporter-8m9g4" (UID: "e74848c0-7cd6-4754-936c-e4733c3764f8") : secret "node-exporter-tls" not found Jul 09 18:26:59.058050 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.057920 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-textfile\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.058050 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.057932 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e74848c0-7cd6-4754-936c-e4733c3764f8-sys\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.058050 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.057957 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e74848c0-7cd6-4754-936c-e4733c3764f8-root\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.058200 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.058167 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.058334 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.058315 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e74848c0-7cd6-4754-936c-e4733c3764f8-metrics-client-ca\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.060005 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.059982 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.065997 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.065977 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd4cv\" (UniqueName: \"kubernetes.io/projected/e74848c0-7cd6-4754-936c-e4733c3764f8-kube-api-access-kd4cv\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.109237 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.109204 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:59.210279 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.210246 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:59.281094 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.281010 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:59.563000 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.562932 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-tls\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.565096 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.565073 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e74848c0-7cd6-4754-936c-e4733c3764f8-node-exporter-tls\") pod \"node-exporter-8m9g4\" (UID: \"e74848c0-7cd6-4754-936c-e4733c3764f8\") " pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.579977 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.579949 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:59.724805 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.724770 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:59.750810 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.750784 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8m9g4" Jul 09 18:26:59.762586 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:26:59.762552 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74848c0_7cd6_4754_936c_e4733c3764f8.slice/crio-13fbe2f36203f60da2a9c75769b14cf2d6cfa178ce4154416c6cd670ed817484 WatchSource:0}: Error finding container 13fbe2f36203f60da2a9c75769b14cf2d6cfa178ce4154416c6cd670ed817484: Status 404 returned error can't find the container with id 13fbe2f36203f60da2a9c75769b14cf2d6cfa178ce4154416c6cd670ed817484 Jul 09 18:26:59.817996 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.817891 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:26:59.877844 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.877815 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4txx9" Jul 09 18:26:59.906565 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.906531 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8m9g4" event={"ID":"e74848c0-7cd6-4754-936c-e4733c3764f8","Type":"ContainerStarted","Data":"13fbe2f36203f60da2a9c75769b14cf2d6cfa178ce4154416c6cd670ed817484"} Jul 09 18:26:59.906830 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:26:59.906804 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:00.178239 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:00.178205 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:00.281211 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:00.281181 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:01.905945 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:01.905907 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:01.915862 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:01.915830 2558 generic.go:356] "Generic (PLEG): container finished" podID="e74848c0-7cd6-4754-936c-e4733c3764f8" containerID="911755b9dbf4aafed79aa9342f0b5a2c54a86ec876b2670294762cb77839cac4" exitCode=0 Jul 09 18:27:01.916008 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:01.915920 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8m9g4" event={"ID":"e74848c0-7cd6-4754-936c-e4733c3764f8","Type":"ContainerDied","Data":"911755b9dbf4aafed79aa9342f0b5a2c54a86ec876b2670294762cb77839cac4"} Jul 09 18:27:01.996129 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:01.996093 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:02.103846 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:02.103819 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:02.248083 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:02.247986 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:02.821444 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:02.821395 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:02.920937 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:02.920902 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:02.921581 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:02.921559 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8m9g4" event={"ID":"e74848c0-7cd6-4754-936c-e4733c3764f8","Type":"ContainerStarted","Data":"f01945df73f374f685544ebebecf3276506b225260cecfe87252e6b21c4f7c60"} Jul 09 18:27:02.921694 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:02.921612 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8m9g4" event={"ID":"e74848c0-7cd6-4754-936c-e4733c3764f8","Type":"ContainerStarted","Data":"f863842cd35b9d175e1ca9a8d0e67100c7172ca7492150c16e89d7f476906471"} Jul 09 18:27:02.952901 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:02.952812 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8m9g4" podStartSLOduration=3.680948705 podStartE2EDuration="4.952794017s" totalImagesPullingTime="1.208303468s" totalInitContainerRuntime="63.541844ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:27:02.951848502 +0000 UTC m=+90.016514177" watchObservedRunningTime="2026-07-09 18:27:02.952794017 +0000 UTC m=+90.017459691" Jul 09 18:27:03.201816 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:03.201783 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:03.298910 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:03.298879 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:03.516016 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:03.515940 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:03.640739 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:03.640686 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:03.836744 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:03.836644 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:03.942071 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:03.942036 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:04.064013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.063982 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7b594d457f-nkgsk"] Jul 09 18:27:04.067389 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.067365 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.070747 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.070728 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 09 18:27:04.070872 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.070855 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 09 18:27:04.070941 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.070920 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 09 18:27:04.070996 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.070946 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 09 18:27:04.071081 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.071046 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kkqhq\"" Jul 09 18:27:04.071282 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.071265 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 09 18:27:04.076999 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.076981 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 09 18:27:04.083471 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.083446 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b594d457f-nkgsk"] Jul 09 18:27:04.200635 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.200596 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d5805c73-6290-41e0-8e37-4449efb07bce-secret-telemeter-client\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.200635 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.200634 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5805c73-6290-41e0-8e37-4449efb07bce-serving-certs-ca-bundle\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.200878 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.200664 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fgzk\" (UniqueName: \"kubernetes.io/projected/d5805c73-6290-41e0-8e37-4449efb07bce-kube-api-access-8fgzk\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.200878 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.200800 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5805c73-6290-41e0-8e37-4449efb07bce-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.200878 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.200855 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5805c73-6290-41e0-8e37-4449efb07bce-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.200975 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.200878 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d5805c73-6290-41e0-8e37-4449efb07bce-telemeter-client-tls\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.200975 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.200895 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d5805c73-6290-41e0-8e37-4449efb07bce-federate-client-tls\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.200975 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.200917 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5805c73-6290-41e0-8e37-4449efb07bce-metrics-client-ca\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.301383 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.301349 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5805c73-6290-41e0-8e37-4449efb07bce-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.301544 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.301388 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d5805c73-6290-41e0-8e37-4449efb07bce-telemeter-client-tls\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.301544 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.301408 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d5805c73-6290-41e0-8e37-4449efb07bce-federate-client-tls\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.301544 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.301425 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5805c73-6290-41e0-8e37-4449efb07bce-metrics-client-ca\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.301544 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.301446 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d5805c73-6290-41e0-8e37-4449efb07bce-secret-telemeter-client\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.301544 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.301470 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5805c73-6290-41e0-8e37-4449efb07bce-serving-certs-ca-bundle\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.301544 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.301488 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fgzk\" (UniqueName: \"kubernetes.io/projected/d5805c73-6290-41e0-8e37-4449efb07bce-kube-api-access-8fgzk\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.301544 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.301529 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5805c73-6290-41e0-8e37-4449efb07bce-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.302356 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.302325 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5805c73-6290-41e0-8e37-4449efb07bce-serving-certs-ca-bundle\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.302617 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.302589 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5805c73-6290-41e0-8e37-4449efb07bce-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.302766 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.302745 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5805c73-6290-41e0-8e37-4449efb07bce-metrics-client-ca\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.304059 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.304036 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d5805c73-6290-41e0-8e37-4449efb07bce-secret-telemeter-client\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.304162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.304070 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d5805c73-6290-41e0-8e37-4449efb07bce-federate-client-tls\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.304216 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.304193 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5805c73-6290-41e0-8e37-4449efb07bce-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.304276 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.304241 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d5805c73-6290-41e0-8e37-4449efb07bce-telemeter-client-tls\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.312567 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.312544 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fgzk\" (UniqueName: \"kubernetes.io/projected/d5805c73-6290-41e0-8e37-4449efb07bce-kube-api-access-8fgzk\") pod \"telemeter-client-7b594d457f-nkgsk\" (UID: \"d5805c73-6290-41e0-8e37-4449efb07bce\") " pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.376757 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.376725 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" Jul 09 18:27:04.514383 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.514342 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b594d457f-nkgsk"] Jul 09 18:27:04.519423 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:27:04.519395 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5805c73_6290_41e0_8e37_4449efb07bce.slice/crio-0f3db1a3361acb065851f8892b7b364d486dfcaadbfd63129c521375987b09e3 WatchSource:0}: Error finding container 0f3db1a3361acb065851f8892b7b364d486dfcaadbfd63129c521375987b09e3: Status 404 returned error can't find the container with id 0f3db1a3361acb065851f8892b7b364d486dfcaadbfd63129c521375987b09e3 Jul 09 18:27:04.930149 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:04.930113 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" event={"ID":"d5805c73-6290-41e0-8e37-4449efb07bce","Type":"ContainerStarted","Data":"0f3db1a3361acb065851f8892b7b364d486dfcaadbfd63129c521375987b09e3"} Jul 09 18:27:05.250919 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:05.250794 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:05.387636 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:05.387597 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:05.504162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:05.504087 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:07.942575 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:07.942534 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" event={"ID":"d5805c73-6290-41e0-8e37-4449efb07bce","Type":"ContainerStarted","Data":"248108c084dc042948f43712bc78ae12ed8ceb9a5a9fab124bef96eff2a4a54d"} Jul 09 18:27:08.886221 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:08.886185 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:08.948310 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:08.948273 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" event={"ID":"d5805c73-6290-41e0-8e37-4449efb07bce","Type":"ContainerStarted","Data":"a73c83cbff4b7706c87950f9a120ace1b5cb4bdbdbb39e78161c735116a89c21"} Jul 09 18:27:08.948666 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:08.948339 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" event={"ID":"d5805c73-6290-41e0-8e37-4449efb07bce","Type":"ContainerStarted","Data":"4d2a5ea98794fc527e066618b0ab737c2ce7818ae032bb4de1ecb75dd9ab6a72"} Jul 09 18:27:08.983962 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:08.983911 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7b594d457f-nkgsk" podStartSLOduration=1.215039336 podStartE2EDuration="4.983895477s" totalImagesPullingTime="3.768856141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:27:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:27:08.981404376 +0000 UTC m=+96.046070050" watchObservedRunningTime="2026-07-09 18:27:08.983895477 +0000 UTC m=+96.048561173" Jul 09 18:27:09.470772 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:09.470740 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:09.565076 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:09.565045 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:10.771475 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:10.771443 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:10.838252 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:10.838217 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:12.426011 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:12.425968 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:12.528062 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:12.528029 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:16.379004 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:16.378972 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:16.479203 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:16.479168 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:16.532016 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:16.531979 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:20.878933 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:20.878904 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j2xsk" Jul 09 18:27:22.950268 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:22.950236 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:23.050133 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:23.050098 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:23.106281 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:23.106250 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:24.422775 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:24.422738 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:24.481658 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:24.481622 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:36.109335 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:36.109303 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:36.163090 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:36.163059 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:27:48.085871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:48.085836 2558 generic.go:356] "Generic (PLEG): container finished" podID="a3b826ae-4a36-452c-a569-1a7cdbb40b06" containerID="b1ffefd0e6989aa5ae1fb07e59c62f3230a470cb4cc45c9ebe22dc1a723a8430" exitCode=0 Jul 09 18:27:48.086291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:48.085905 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" event={"ID":"a3b826ae-4a36-452c-a569-1a7cdbb40b06","Type":"ContainerDied","Data":"b1ffefd0e6989aa5ae1fb07e59c62f3230a470cb4cc45c9ebe22dc1a723a8430"} Jul 09 18:27:48.086291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:48.086235 2558 scope.go:117] "RemoveContainer" containerID="b1ffefd0e6989aa5ae1fb07e59c62f3230a470cb4cc45c9ebe22dc1a723a8430" Jul 09 18:27:49.091512 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:27:49.091480 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-v64gl" event={"ID":"a3b826ae-4a36-452c-a569-1a7cdbb40b06","Type":"ContainerStarted","Data":"52ceb6d70ea0651677d21c33186fb37eda1d0ff41ab74aae3a167455c6dacf06"} Jul 09 18:28:13.556670 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:28:13.556632 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:15.112989 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:28:15.112952 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:15.174448 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:28:15.174409 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:27.019592 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:28:27.019511 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:27.116062 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:28:27.116021 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:39.443207 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:28:39.443170 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:39.543269 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:28:39.543239 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:39.602576 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:28:39.602543 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:40.430092 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:28:40.430033 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:40.498555 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:28:40.498523 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:51.462966 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:28:51.462932 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:51.564536 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:28:51.564503 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:51.644807 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:28:51.644773 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:00.779103 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:00.779072 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:00.873881 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:00.873848 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:20.108237 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:20.108189 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:20.162274 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:20.162245 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:22.275962 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:22.275928 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:22.572369 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:22.572281 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:24.205619 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:24.205587 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4m5jj"] Jul 09 18:29:24.208845 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:24.208829 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4m5jj" Jul 09 18:29:24.211326 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:24.211303 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-djnqk\"" Jul 09 18:29:24.211544 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:24.211523 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:29:24.211640 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:24.211555 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 09 18:29:24.221519 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:24.221495 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/620cb139-b0c8-40b0-bab4-979a8dea8de9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-4m5jj\" (UID: \"620cb139-b0c8-40b0-bab4-979a8dea8de9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4m5jj" Jul 09 18:29:24.221647 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:24.221540 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6t5r\" (UniqueName: \"kubernetes.io/projected/620cb139-b0c8-40b0-bab4-979a8dea8de9-kube-api-access-b6t5r\") pod \"cert-manager-operator-controller-manager-55c444bf86-4m5jj\" (UID: \"620cb139-b0c8-40b0-bab4-979a8dea8de9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4m5jj" Jul 09 18:29:24.223110 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:24.223090 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4m5jj"] Jul 09 18:29:24.322411 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:24.322372 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/620cb139-b0c8-40b0-bab4-979a8dea8de9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-4m5jj\" (UID: \"620cb139-b0c8-40b0-bab4-979a8dea8de9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4m5jj" Jul 09 18:29:24.322411 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:24.322424 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6t5r\" (UniqueName: \"kubernetes.io/projected/620cb139-b0c8-40b0-bab4-979a8dea8de9-kube-api-access-b6t5r\") pod \"cert-manager-operator-controller-manager-55c444bf86-4m5jj\" (UID: \"620cb139-b0c8-40b0-bab4-979a8dea8de9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4m5jj" Jul 09 18:29:24.322843 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:24.322817 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/620cb139-b0c8-40b0-bab4-979a8dea8de9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-4m5jj\" (UID: \"620cb139-b0c8-40b0-bab4-979a8dea8de9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4m5jj" Jul 09 18:29:24.334891 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:24.334858 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6t5r\" (UniqueName: \"kubernetes.io/projected/620cb139-b0c8-40b0-bab4-979a8dea8de9-kube-api-access-b6t5r\") pod \"cert-manager-operator-controller-manager-55c444bf86-4m5jj\" (UID: \"620cb139-b0c8-40b0-bab4-979a8dea8de9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4m5jj" Jul 09 18:29:24.517784 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:24.517652 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4m5jj" Jul 09 18:29:24.671400 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:24.671372 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4m5jj"] Jul 09 18:29:24.674400 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:29:24.674370 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620cb139_b0c8_40b0_bab4_979a8dea8de9.slice/crio-736109f5a7e0a208cd55cbb9ca0468f46e5308ce417cd03fdcb51b07de98b614 WatchSource:0}: Error finding container 736109f5a7e0a208cd55cbb9ca0468f46e5308ce417cd03fdcb51b07de98b614: Status 404 returned error can't find the container with id 736109f5a7e0a208cd55cbb9ca0468f46e5308ce417cd03fdcb51b07de98b614 Jul 09 18:29:25.183923 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:25.183874 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:25.427444 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:25.427407 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4m5jj" event={"ID":"620cb139-b0c8-40b0-bab4-979a8dea8de9","Type":"ContainerStarted","Data":"736109f5a7e0a208cd55cbb9ca0468f46e5308ce417cd03fdcb51b07de98b614"} Jul 09 18:29:25.870864 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:25.870824 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:25.971648 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:25.971613 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:27.437461 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:27.437381 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4m5jj" event={"ID":"620cb139-b0c8-40b0-bab4-979a8dea8de9","Type":"ContainerStarted","Data":"da1c7dee1613bdcc38c221fc788b860ba3cb86b0282a126b2435c38134f51f53"} Jul 09 18:29:27.459332 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:27.459287 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4m5jj" podStartSLOduration=0.97651754 podStartE2EDuration="3.459268453s" totalImagesPullingTime="2.482750913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:29:27.458017585 +0000 UTC m=+234.522683259" watchObservedRunningTime="2026-07-09 18:29:27.459268453 +0000 UTC m=+234.523934127" Jul 09 18:29:27.866066 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:27.865977 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:27.958564 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:27.958527 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:27.999783 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:27.999746 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94"] Jul 09 18:29:28.003416 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.003384 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" Jul 09 18:29:28.009226 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.009203 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5rb4f\"" Jul 09 18:29:28.009493 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.009266 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 18:29:28.010220 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.010196 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 18:29:28.020515 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.020492 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94"] Jul 09 18:29:28.051538 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.051509 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3652fc69-186f-4469-bfaf-422ea3da91c0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94\" (UID: \"3652fc69-186f-4469-bfaf-422ea3da91c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" Jul 09 18:29:28.051679 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.051553 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r78mf\" (UniqueName: \"kubernetes.io/projected/3652fc69-186f-4469-bfaf-422ea3da91c0-kube-api-access-r78mf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94\" (UID: \"3652fc69-186f-4469-bfaf-422ea3da91c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" Jul 09 18:29:28.051796 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.051770 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3652fc69-186f-4469-bfaf-422ea3da91c0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94\" (UID: \"3652fc69-186f-4469-bfaf-422ea3da91c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" Jul 09 18:29:28.086621 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.086593 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:28.152922 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.152893 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3652fc69-186f-4469-bfaf-422ea3da91c0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94\" (UID: \"3652fc69-186f-4469-bfaf-422ea3da91c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" Jul 09 18:29:28.153060 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.152932 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3652fc69-186f-4469-bfaf-422ea3da91c0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94\" (UID: \"3652fc69-186f-4469-bfaf-422ea3da91c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" Jul 09 18:29:28.153060 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.152958 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r78mf\" (UniqueName: \"kubernetes.io/projected/3652fc69-186f-4469-bfaf-422ea3da91c0-kube-api-access-r78mf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94\" (UID: \"3652fc69-186f-4469-bfaf-422ea3da91c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" Jul 09 18:29:28.153314 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.153292 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3652fc69-186f-4469-bfaf-422ea3da91c0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94\" (UID: \"3652fc69-186f-4469-bfaf-422ea3da91c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" Jul 09 18:29:28.153383 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.153349 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3652fc69-186f-4469-bfaf-422ea3da91c0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94\" (UID: \"3652fc69-186f-4469-bfaf-422ea3da91c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" Jul 09 18:29:28.171390 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.171362 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r78mf\" (UniqueName: \"kubernetes.io/projected/3652fc69-186f-4469-bfaf-422ea3da91c0-kube-api-access-r78mf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94\" (UID: \"3652fc69-186f-4469-bfaf-422ea3da91c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" Jul 09 18:29:28.192279 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.192215 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:28.315356 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.315325 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" Jul 09 18:29:28.448929 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:28.448851 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94"] Jul 09 18:29:28.451693 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:29:28.451659 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3652fc69_186f_4469_bfaf_422ea3da91c0.slice/crio-376f00f59ca555d70deda749e43568934e1835ddd9baedbf0cd964d220c2513f WatchSource:0}: Error finding container 376f00f59ca555d70deda749e43568934e1835ddd9baedbf0cd964d220c2513f: Status 404 returned error can't find the container with id 376f00f59ca555d70deda749e43568934e1835ddd9baedbf0cd964d220c2513f Jul 09 18:29:29.446325 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:29.446290 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" event={"ID":"3652fc69-186f-4469-bfaf-422ea3da91c0","Type":"ContainerStarted","Data":"376f00f59ca555d70deda749e43568934e1835ddd9baedbf0cd964d220c2513f"} Jul 09 18:29:30.484868 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:30.484830 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:30.886235 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:30.886184 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:32.641731 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:32.641672 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:32.695248 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:32.695209 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:36.477843 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:36.477806 2558 generic.go:356] "Generic (PLEG): container finished" podID="3652fc69-186f-4469-bfaf-422ea3da91c0" containerID="4a844d7e56fd86bf2335cf076aedec7bb1094b50a72288f7f761abb6f873dff8" exitCode=0 Jul 09 18:29:36.478238 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:36.477853 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" event={"ID":"3652fc69-186f-4469-bfaf-422ea3da91c0","Type":"ContainerDied","Data":"4a844d7e56fd86bf2335cf076aedec7bb1094b50a72288f7f761abb6f873dff8"} Jul 09 18:29:38.673493 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:38.673465 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:39.491271 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:39.491234 2558 generic.go:356] "Generic (PLEG): container finished" podID="3652fc69-186f-4469-bfaf-422ea3da91c0" containerID="69df4a309c8cd068067fabc3f1f17bfe228b01138b7e2fe260319665eb703cba" exitCode=0 Jul 09 18:29:39.491466 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:39.491320 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" event={"ID":"3652fc69-186f-4469-bfaf-422ea3da91c0","Type":"ContainerDied","Data":"69df4a309c8cd068067fabc3f1f17bfe228b01138b7e2fe260319665eb703cba"} Jul 09 18:29:42.645471 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:42.645433 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:48.529813 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:48.529721 2558 generic.go:356] "Generic (PLEG): container finished" podID="3652fc69-186f-4469-bfaf-422ea3da91c0" containerID="9272260cc034edf6c8c4c1755fd6b51d4de740707100fcd3b4472b906f9010f0" exitCode=0 Jul 09 18:29:48.529813 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:48.529776 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" event={"ID":"3652fc69-186f-4469-bfaf-422ea3da91c0","Type":"ContainerDied","Data":"9272260cc034edf6c8c4c1755fd6b51d4de740707100fcd3b4472b906f9010f0"} Jul 09 18:29:49.657164 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:49.657139 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" Jul 09 18:29:49.732110 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:49.732078 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3652fc69-186f-4469-bfaf-422ea3da91c0-bundle\") pod \"3652fc69-186f-4469-bfaf-422ea3da91c0\" (UID: \"3652fc69-186f-4469-bfaf-422ea3da91c0\") " Jul 09 18:29:49.732273 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:49.732170 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3652fc69-186f-4469-bfaf-422ea3da91c0-util\") pod \"3652fc69-186f-4469-bfaf-422ea3da91c0\" (UID: \"3652fc69-186f-4469-bfaf-422ea3da91c0\") " Jul 09 18:29:49.732273 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:49.732207 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r78mf\" (UniqueName: \"kubernetes.io/projected/3652fc69-186f-4469-bfaf-422ea3da91c0-kube-api-access-r78mf\") pod \"3652fc69-186f-4469-bfaf-422ea3da91c0\" (UID: \"3652fc69-186f-4469-bfaf-422ea3da91c0\") " Jul 09 18:29:49.732462 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:49.732438 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3652fc69-186f-4469-bfaf-422ea3da91c0-bundle" (OuterVolumeSpecName: "bundle") pod "3652fc69-186f-4469-bfaf-422ea3da91c0" (UID: "3652fc69-186f-4469-bfaf-422ea3da91c0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:29:49.734504 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:49.734483 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3652fc69-186f-4469-bfaf-422ea3da91c0-kube-api-access-r78mf" (OuterVolumeSpecName: "kube-api-access-r78mf") pod "3652fc69-186f-4469-bfaf-422ea3da91c0" (UID: "3652fc69-186f-4469-bfaf-422ea3da91c0"). InnerVolumeSpecName "kube-api-access-r78mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:29:49.737545 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:49.737524 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3652fc69-186f-4469-bfaf-422ea3da91c0-util" (OuterVolumeSpecName: "util") pod "3652fc69-186f-4469-bfaf-422ea3da91c0" (UID: "3652fc69-186f-4469-bfaf-422ea3da91c0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:29:49.833260 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:49.833183 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3652fc69-186f-4469-bfaf-422ea3da91c0-util\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:29:49.833260 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:49.833209 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r78mf\" (UniqueName: \"kubernetes.io/projected/3652fc69-186f-4469-bfaf-422ea3da91c0-kube-api-access-r78mf\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:29:49.833260 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:49.833219 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3652fc69-186f-4469-bfaf-422ea3da91c0-bundle\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:29:50.539713 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:50.539669 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" Jul 09 18:29:50.539713 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:50.539672 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg6x94" event={"ID":"3652fc69-186f-4469-bfaf-422ea3da91c0","Type":"ContainerDied","Data":"376f00f59ca555d70deda749e43568934e1835ddd9baedbf0cd964d220c2513f"} Jul 09 18:29:50.539951 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:50.539746 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="376f00f59ca555d70deda749e43568934e1835ddd9baedbf0cd964d220c2513f" Jul 09 18:29:55.185254 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:55.185174 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:55.282191 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:55.282163 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:55.512814 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:55.512726 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:55.614628 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:29:55.614592 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:01.196769 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:01.196729 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:01.351202 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:01.351168 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:03.393593 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:03.393563 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:14.930687 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:14.930653 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:15.030779 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.030745 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:15.549077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.549040 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:15.692825 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.692795 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:15.791847 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.791816 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:15.815299 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.815215 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v"] Jul 09 18:30:15.815476 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.815465 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3652fc69-186f-4469-bfaf-422ea3da91c0" containerName="util" Jul 09 18:30:15.815517 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.815478 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3652fc69-186f-4469-bfaf-422ea3da91c0" containerName="util" Jul 09 18:30:15.815517 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.815492 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3652fc69-186f-4469-bfaf-422ea3da91c0" containerName="pull" Jul 09 18:30:15.815517 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.815497 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3652fc69-186f-4469-bfaf-422ea3da91c0" containerName="pull" Jul 09 18:30:15.815517 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.815507 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3652fc69-186f-4469-bfaf-422ea3da91c0" containerName="extract" Jul 09 18:30:15.815517 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.815513 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3652fc69-186f-4469-bfaf-422ea3da91c0" containerName="extract" Jul 09 18:30:15.815660 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.815557 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3652fc69-186f-4469-bfaf-422ea3da91c0" containerName="extract" Jul 09 18:30:15.818444 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.818429 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:15.822006 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.821987 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 18:30:15.822356 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.822334 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 18:30:15.822625 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.822605 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 18:30:15.822859 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.822838 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 18:30:15.822970 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.822932 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fmrpn\"" Jul 09 18:30:15.838961 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.838938 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v"] Jul 09 18:30:15.934665 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.934634 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9ebdde9-5ed0-4512-9e4b-da303966d500-webhook-cert\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:15.934665 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.934668 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9ebdde9-5ed0-4512-9e4b-da303966d500-tmp\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:15.935098 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.934723 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb94c\" (UniqueName: \"kubernetes.io/projected/f9ebdde9-5ed0-4512-9e4b-da303966d500-kube-api-access-rb94c\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:15.935098 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.934809 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9ebdde9-5ed0-4512-9e4b-da303966d500-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:15.935098 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.934857 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9ebdde9-5ed0-4512-9e4b-da303966d500-manifests\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:15.952790 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:15.952761 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:16.035415 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.035378 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb94c\" (UniqueName: \"kubernetes.io/projected/f9ebdde9-5ed0-4512-9e4b-da303966d500-kube-api-access-rb94c\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:16.035558 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.035437 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9ebdde9-5ed0-4512-9e4b-da303966d500-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:16.035558 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.035474 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9ebdde9-5ed0-4512-9e4b-da303966d500-manifests\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:16.035558 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.035508 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9ebdde9-5ed0-4512-9e4b-da303966d500-webhook-cert\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:16.035558 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.035524 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9ebdde9-5ed0-4512-9e4b-da303966d500-tmp\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:16.036003 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.035973 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9ebdde9-5ed0-4512-9e4b-da303966d500-manifests\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:16.037674 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.037650 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9ebdde9-5ed0-4512-9e4b-da303966d500-tmp\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:16.038150 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.038130 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9ebdde9-5ed0-4512-9e4b-da303966d500-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:16.038198 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.038130 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9ebdde9-5ed0-4512-9e4b-da303966d500-webhook-cert\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:16.056365 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.056332 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:16.059932 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.059911 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb94c\" (UniqueName: \"kubernetes.io/projected/f9ebdde9-5ed0-4512-9e4b-da303966d500-kube-api-access-rb94c\") pod \"opendatahub-operator-controller-manager-59f7885b79-z4m6v\" (UID: \"f9ebdde9-5ed0-4512-9e4b-da303966d500\") " pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:16.127833 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.127767 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:16.168461 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.168424 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:16.267189 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.267147 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:16.268263 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.268238 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v"] Jul 09 18:30:16.271392 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:30:16.271371 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ebdde9_5ed0_4512_9e4b_da303966d500.slice/crio-ba3ae90d532893ffb0c454956aae476c801f53eadae60694fded60e91d48e8de WatchSource:0}: Error finding container ba3ae90d532893ffb0c454956aae476c801f53eadae60694fded60e91d48e8de: Status 404 returned error can't find the container with id ba3ae90d532893ffb0c454956aae476c801f53eadae60694fded60e91d48e8de Jul 09 18:30:16.353690 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.353656 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:16.454146 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.454108 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:16.554077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.554045 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:16.637031 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:16.636989 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" event={"ID":"f9ebdde9-5ed0-4512-9e4b-da303966d500","Type":"ContainerStarted","Data":"ba3ae90d532893ffb0c454956aae476c801f53eadae60694fded60e91d48e8de"} Jul 09 18:30:19.651269 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:19.651183 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" event={"ID":"f9ebdde9-5ed0-4512-9e4b-da303966d500","Type":"ContainerStarted","Data":"675e50fe7e8e03794130b130133418d34fe1e4c288ff701b95509cc02a6247e9"} Jul 09 18:30:20.657160 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:20.657123 2558 generic.go:356] "Generic (PLEG): container finished" podID="f9ebdde9-5ed0-4512-9e4b-da303966d500" containerID="675e50fe7e8e03794130b130133418d34fe1e4c288ff701b95509cc02a6247e9" exitCode=0 Jul 09 18:30:20.657517 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:20.657215 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" event={"ID":"f9ebdde9-5ed0-4512-9e4b-da303966d500","Type":"ContainerDied","Data":"675e50fe7e8e03794130b130133418d34fe1e4c288ff701b95509cc02a6247e9"} Jul 09 18:30:21.663582 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:21.663542 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" event={"ID":"f9ebdde9-5ed0-4512-9e4b-da303966d500","Type":"ContainerStarted","Data":"31156c0d64495e3b5bd8e6c9e74d3054d824651f971c49eb2efe6702f830d9ff"} Jul 09 18:30:21.664084 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:21.663648 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:21.674509 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:21.674476 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:21.721072 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:21.721015 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" podStartSLOduration=2.120421582 podStartE2EDuration="6.720998338s" totalImagesPullingTime="3.499305656s" totalInitContainerRuntime="1.1012711s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:30:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:30:21.720603491 +0000 UTC m=+288.785269169" watchObservedRunningTime="2026-07-09 18:30:21.720998338 +0000 UTC m=+288.785664049" Jul 09 18:30:21.775614 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:21.775580 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:32.671225 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:32.671191 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-59f7885b79-z4m6v" Jul 09 18:30:33.279311 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:33.279275 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:30:33.284177 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:33.284147 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:30:33.289212 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:33.289196 2558 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 18:30:34.755599 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:34.755542 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:35.594935 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.594898 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq"] Jul 09 18:30:35.601821 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.601786 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq" Jul 09 18:30:35.605200 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.605180 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 18:30:35.605346 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.605227 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 18:30:35.620188 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.620166 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq"] Jul 09 18:30:35.688494 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.688461 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx7vt\" (UniqueName: \"kubernetes.io/projected/f5cb28e7-4f29-4d75-b227-6efd0e1db6f5-kube-api-access-rx7vt\") pod \"kube-auth-proxy-668f4b5f87-c8cgq\" (UID: \"f5cb28e7-4f29-4d75-b227-6efd0e1db6f5\") " pod="openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq" Jul 09 18:30:35.688648 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.688506 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5cb28e7-4f29-4d75-b227-6efd0e1db6f5-tls-certs\") pod \"kube-auth-proxy-668f4b5f87-c8cgq\" (UID: \"f5cb28e7-4f29-4d75-b227-6efd0e1db6f5\") " pod="openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq" Jul 09 18:30:35.688648 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.688523 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5cb28e7-4f29-4d75-b227-6efd0e1db6f5-tmp\") pod \"kube-auth-proxy-668f4b5f87-c8cgq\" (UID: \"f5cb28e7-4f29-4d75-b227-6efd0e1db6f5\") " pod="openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq" Jul 09 18:30:35.789842 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.789811 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5cb28e7-4f29-4d75-b227-6efd0e1db6f5-tls-certs\") pod \"kube-auth-proxy-668f4b5f87-c8cgq\" (UID: \"f5cb28e7-4f29-4d75-b227-6efd0e1db6f5\") " pod="openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq" Jul 09 18:30:35.789842 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.789842 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5cb28e7-4f29-4d75-b227-6efd0e1db6f5-tmp\") pod \"kube-auth-proxy-668f4b5f87-c8cgq\" (UID: \"f5cb28e7-4f29-4d75-b227-6efd0e1db6f5\") " pod="openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq" Jul 09 18:30:35.790259 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.789898 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx7vt\" (UniqueName: \"kubernetes.io/projected/f5cb28e7-4f29-4d75-b227-6efd0e1db6f5-kube-api-access-rx7vt\") pod \"kube-auth-proxy-668f4b5f87-c8cgq\" (UID: \"f5cb28e7-4f29-4d75-b227-6efd0e1db6f5\") " pod="openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq" Jul 09 18:30:35.792149 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.792120 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5cb28e7-4f29-4d75-b227-6efd0e1db6f5-tmp\") pod \"kube-auth-proxy-668f4b5f87-c8cgq\" (UID: \"f5cb28e7-4f29-4d75-b227-6efd0e1db6f5\") " pod="openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq" Jul 09 18:30:35.792267 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.792247 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5cb28e7-4f29-4d75-b227-6efd0e1db6f5-tls-certs\") pod \"kube-auth-proxy-668f4b5f87-c8cgq\" (UID: \"f5cb28e7-4f29-4d75-b227-6efd0e1db6f5\") " pod="openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq" Jul 09 18:30:35.843166 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.843138 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pth4s"] Jul 09 18:30:35.843953 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.843932 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx7vt\" (UniqueName: \"kubernetes.io/projected/f5cb28e7-4f29-4d75-b227-6efd0e1db6f5-kube-api-access-rx7vt\") pod \"kube-auth-proxy-668f4b5f87-c8cgq\" (UID: \"f5cb28e7-4f29-4d75-b227-6efd0e1db6f5\") " pod="openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq" Jul 09 18:30:35.846306 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.846268 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pth4s" Jul 09 18:30:35.855870 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.855848 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pth4s"] Jul 09 18:30:35.855978 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.855850 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 09 18:30:35.864034 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.864017 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-x4ncf\"" Jul 09 18:30:35.891253 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.891225 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8q42\" (UniqueName: \"kubernetes.io/projected/3fddb4de-381b-4b3a-8e5c-c7d1462f2645-kube-api-access-q8q42\") pod \"odh-model-controller-7998c67dc4-pth4s\" (UID: \"3fddb4de-381b-4b3a-8e5c-c7d1462f2645\") " pod="opendatahub/odh-model-controller-7998c67dc4-pth4s" Jul 09 18:30:35.891339 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.891321 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fddb4de-381b-4b3a-8e5c-c7d1462f2645-cert\") pod \"odh-model-controller-7998c67dc4-pth4s\" (UID: \"3fddb4de-381b-4b3a-8e5c-c7d1462f2645\") " pod="opendatahub/odh-model-controller-7998c67dc4-pth4s" Jul 09 18:30:35.911139 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.911116 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq" Jul 09 18:30:35.992852 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.992503 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fddb4de-381b-4b3a-8e5c-c7d1462f2645-cert\") pod \"odh-model-controller-7998c67dc4-pth4s\" (UID: \"3fddb4de-381b-4b3a-8e5c-c7d1462f2645\") " pod="opendatahub/odh-model-controller-7998c67dc4-pth4s" Jul 09 18:30:35.992852 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:35.992540 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8q42\" (UniqueName: \"kubernetes.io/projected/3fddb4de-381b-4b3a-8e5c-c7d1462f2645-kube-api-access-q8q42\") pod \"odh-model-controller-7998c67dc4-pth4s\" (UID: \"3fddb4de-381b-4b3a-8e5c-c7d1462f2645\") " pod="opendatahub/odh-model-controller-7998c67dc4-pth4s" Jul 09 18:30:35.992852 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:30:35.992660 2558 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 09 18:30:35.992852 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:30:35.992744 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fddb4de-381b-4b3a-8e5c-c7d1462f2645-cert podName:3fddb4de-381b-4b3a-8e5c-c7d1462f2645 nodeName:}" failed. No retries permitted until 2026-07-09 18:30:36.492727645 +0000 UTC m=+303.557393297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3fddb4de-381b-4b3a-8e5c-c7d1462f2645-cert") pod "odh-model-controller-7998c67dc4-pth4s" (UID: "3fddb4de-381b-4b3a-8e5c-c7d1462f2645") : secret "odh-model-controller-webhook-cert" not found Jul 09 18:30:36.016639 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.016578 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8q42\" (UniqueName: \"kubernetes.io/projected/3fddb4de-381b-4b3a-8e5c-c7d1462f2645-kube-api-access-q8q42\") pod \"odh-model-controller-7998c67dc4-pth4s\" (UID: \"3fddb4de-381b-4b3a-8e5c-c7d1462f2645\") " pod="opendatahub/odh-model-controller-7998c67dc4-pth4s" Jul 09 18:30:36.133234 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:30:36.133198 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb28e7_4f29_4d75_b227_6efd0e1db6f5.slice/crio-72463650ba6b51347267708c514348cdb849a925ae0aae7d43123ef82ed1e9f8 WatchSource:0}: Error finding container 72463650ba6b51347267708c514348cdb849a925ae0aae7d43123ef82ed1e9f8: Status 404 returned error can't find the container with id 72463650ba6b51347267708c514348cdb849a925ae0aae7d43123ef82ed1e9f8 Jul 09 18:30:36.135571 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.135485 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 18:30:36.135983 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.135956 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq"] Jul 09 18:30:36.404957 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.404927 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:36.497581 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.497544 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fddb4de-381b-4b3a-8e5c-c7d1462f2645-cert\") pod \"odh-model-controller-7998c67dc4-pth4s\" (UID: \"3fddb4de-381b-4b3a-8e5c-c7d1462f2645\") " pod="opendatahub/odh-model-controller-7998c67dc4-pth4s" Jul 09 18:30:36.500200 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.500160 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fddb4de-381b-4b3a-8e5c-c7d1462f2645-cert\") pod \"odh-model-controller-7998c67dc4-pth4s\" (UID: \"3fddb4de-381b-4b3a-8e5c-c7d1462f2645\") " pod="opendatahub/odh-model-controller-7998c67dc4-pth4s" Jul 09 18:30:36.722739 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.722643 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq" event={"ID":"f5cb28e7-4f29-4d75-b227-6efd0e1db6f5","Type":"ContainerStarted","Data":"72463650ba6b51347267708c514348cdb849a925ae0aae7d43123ef82ed1e9f8"} Jul 09 18:30:36.758976 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.758949 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pth4s" Jul 09 18:30:36.831078 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.831044 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52"] Jul 09 18:30:36.835670 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.835645 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:36.842314 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.842285 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:30:36.842660 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.842575 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 18:30:36.842660 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.842606 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 18:30:36.842909 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.842737 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-spnbx\"" Jul 09 18:30:36.843205 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.843080 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 18:30:36.843615 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.843594 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 18:30:36.850559 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.850538 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52"] Jul 09 18:30:36.899940 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.899902 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/faaf27be-52ea-45ec-88a3-fb1259d4f174-metrics-cert\") pod \"lws-controller-manager-576fd7db4f-qxm52\" (UID: \"faaf27be-52ea-45ec-88a3-fb1259d4f174\") " pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:36.900102 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.899972 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/faaf27be-52ea-45ec-88a3-fb1259d4f174-cert\") pod \"lws-controller-manager-576fd7db4f-qxm52\" (UID: \"faaf27be-52ea-45ec-88a3-fb1259d4f174\") " pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:36.900102 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.900038 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/faaf27be-52ea-45ec-88a3-fb1259d4f174-manager-config\") pod \"lws-controller-manager-576fd7db4f-qxm52\" (UID: \"faaf27be-52ea-45ec-88a3-fb1259d4f174\") " pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:36.900102 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.900071 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p86pv\" (UniqueName: \"kubernetes.io/projected/faaf27be-52ea-45ec-88a3-fb1259d4f174-kube-api-access-p86pv\") pod \"lws-controller-manager-576fd7db4f-qxm52\" (UID: \"faaf27be-52ea-45ec-88a3-fb1259d4f174\") " pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:36.942946 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:36.942920 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pth4s"] Jul 09 18:30:36.945891 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:30:36.945858 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fddb4de_381b_4b3a_8e5c_c7d1462f2645.slice/crio-30dd3594a93294ecfdbac5b6700802884c1d4f8085320bb6a77f88b32b94f21a WatchSource:0}: Error finding container 30dd3594a93294ecfdbac5b6700802884c1d4f8085320bb6a77f88b32b94f21a: Status 404 returned error can't find the container with id 30dd3594a93294ecfdbac5b6700802884c1d4f8085320bb6a77f88b32b94f21a Jul 09 18:30:37.001044 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:37.000962 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/faaf27be-52ea-45ec-88a3-fb1259d4f174-cert\") pod \"lws-controller-manager-576fd7db4f-qxm52\" (UID: \"faaf27be-52ea-45ec-88a3-fb1259d4f174\") " pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:37.001044 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:37.001008 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/faaf27be-52ea-45ec-88a3-fb1259d4f174-manager-config\") pod \"lws-controller-manager-576fd7db4f-qxm52\" (UID: \"faaf27be-52ea-45ec-88a3-fb1259d4f174\") " pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:37.001044 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:37.001043 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p86pv\" (UniqueName: \"kubernetes.io/projected/faaf27be-52ea-45ec-88a3-fb1259d4f174-kube-api-access-p86pv\") pod \"lws-controller-manager-576fd7db4f-qxm52\" (UID: \"faaf27be-52ea-45ec-88a3-fb1259d4f174\") " pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:37.001336 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:37.001105 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/faaf27be-52ea-45ec-88a3-fb1259d4f174-metrics-cert\") pod \"lws-controller-manager-576fd7db4f-qxm52\" (UID: \"faaf27be-52ea-45ec-88a3-fb1259d4f174\") " pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:37.001859 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:37.001835 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/faaf27be-52ea-45ec-88a3-fb1259d4f174-manager-config\") pod \"lws-controller-manager-576fd7db4f-qxm52\" (UID: \"faaf27be-52ea-45ec-88a3-fb1259d4f174\") " pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:37.003786 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:37.003763 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/faaf27be-52ea-45ec-88a3-fb1259d4f174-cert\") pod \"lws-controller-manager-576fd7db4f-qxm52\" (UID: \"faaf27be-52ea-45ec-88a3-fb1259d4f174\") " pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:37.003904 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:37.003825 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/faaf27be-52ea-45ec-88a3-fb1259d4f174-metrics-cert\") pod \"lws-controller-manager-576fd7db4f-qxm52\" (UID: \"faaf27be-52ea-45ec-88a3-fb1259d4f174\") " pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:37.027998 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:37.027971 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p86pv\" (UniqueName: \"kubernetes.io/projected/faaf27be-52ea-45ec-88a3-fb1259d4f174-kube-api-access-p86pv\") pod \"lws-controller-manager-576fd7db4f-qxm52\" (UID: \"faaf27be-52ea-45ec-88a3-fb1259d4f174\") " pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:37.147260 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:37.147223 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:37.369352 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:37.369275 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52"] Jul 09 18:30:37.373618 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:30:37.373592 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaaf27be_52ea_45ec_88a3_fb1259d4f174.slice/crio-103b104776f222f858fc51c42d5f9177e47f05253e8190625349253b27ada244 WatchSource:0}: Error finding container 103b104776f222f858fc51c42d5f9177e47f05253e8190625349253b27ada244: Status 404 returned error can't find the container with id 103b104776f222f858fc51c42d5f9177e47f05253e8190625349253b27ada244 Jul 09 18:30:37.693222 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:37.693183 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:37.729779 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:37.729738 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" event={"ID":"faaf27be-52ea-45ec-88a3-fb1259d4f174","Type":"ContainerStarted","Data":"103b104776f222f858fc51c42d5f9177e47f05253e8190625349253b27ada244"} Jul 09 18:30:37.731393 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:37.731329 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pth4s" event={"ID":"3fddb4de-381b-4b3a-8e5c-c7d1462f2645","Type":"ContainerStarted","Data":"30dd3594a93294ecfdbac5b6700802884c1d4f8085320bb6a77f88b32b94f21a"} Jul 09 18:30:37.831309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:37.831271 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:38.395313 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:38.395275 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:38.505882 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:38.505810 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:40.037146 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:40.037112 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:41.755309 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:41.755270 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq" event={"ID":"f5cb28e7-4f29-4d75-b227-6efd0e1db6f5","Type":"ContainerStarted","Data":"1a814efb5b16455f9178e8a851b1e6756065b9f16c48b8cb8af2201eeae554a1"} Jul 09 18:30:41.756879 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:41.756850 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" event={"ID":"faaf27be-52ea-45ec-88a3-fb1259d4f174","Type":"ContainerStarted","Data":"667307b7cc0b0e92175d441ae2dd06779c35e71ff640c10ddfb940427d31fcf6"} Jul 09 18:30:41.756991 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:41.756959 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:30:41.758260 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:41.758238 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pth4s" event={"ID":"3fddb4de-381b-4b3a-8e5c-c7d1462f2645","Type":"ContainerStarted","Data":"06e488bb3271060059ba0633c2a4497b0f9dda79cace0fa39af6c81ec6cd0065"} Jul 09 18:30:41.758322 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:41.758280 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-pth4s" Jul 09 18:30:41.776395 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:41.776289 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-668f4b5f87-c8cgq" podStartSLOduration=2.105426319 podStartE2EDuration="6.77627279s" totalImagesPullingTime="4.670846471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:30:41.775612932 +0000 UTC m=+308.840278606" watchObservedRunningTime="2026-07-09 18:30:41.77627279 +0000 UTC m=+308.840938465" Jul 09 18:30:41.799598 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:41.799549 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" podStartSLOduration=2.372444959 podStartE2EDuration="5.799529575s" totalImagesPullingTime="3.427084616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:30:41.797574363 +0000 UTC m=+308.862240032" watchObservedRunningTime="2026-07-09 18:30:41.799529575 +0000 UTC m=+308.864195249" Jul 09 18:30:41.815767 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:41.815471 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-pth4s" podStartSLOduration=2.946088104 podStartE2EDuration="6.815456327s" totalImagesPullingTime="3.869368223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:30:41.81519364 +0000 UTC m=+308.879859315" watchObservedRunningTime="2026-07-09 18:30:41.815456327 +0000 UTC m=+308.880122001" Jul 09 18:30:44.509946 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:44.509916 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:44.611045 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:44.611008 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:44.926202 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:44.926159 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:45.150179 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:45.150148 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:45.517064 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:45.517022 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:47.301624 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:47.301582 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:47.447477 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:47.447445 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:47.582060 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:47.581961 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:47.707733 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:47.707688 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:48.074186 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:48.074151 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:48.814887 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:48.814853 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:48.910897 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:48.910864 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:49.017202 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:49.017170 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:30:52.765243 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:52.765214 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-pth4s" Jul 09 18:30:52.765757 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:30:52.765391 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-576fd7db4f-qxm52" Jul 09 18:31:36.155319 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:31:36.155244 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:31:36.287592 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:31:36.287554 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:31:36.357871 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:31:36.357839 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:31:37.045163 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:31:37.045132 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:31:37.251502 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:31:37.251469 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:31:42.720588 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:31:42.720558 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:31:57.584652 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:31:57.584613 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:05.634970 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:05.634934 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:06.453180 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:06.453119 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:11.434689 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:11.434653 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:13.182629 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:13.182599 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:13.310522 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:13.310489 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:14.112011 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:14.111978 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:15.332307 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:15.332271 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:17.173607 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:17.173570 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:18.860193 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:18.860151 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:19.593897 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:19.593862 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:21.311674 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:21.311634 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:21.923372 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:21.923336 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:22.061665 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:22.061631 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:22.339262 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:22.339166 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:22.435207 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:22.435176 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:23.061065 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:23.061017 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:23.165011 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:23.164963 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:26.280879 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:26.280844 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:26.379792 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:26.379759 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:27.131934 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:27.131900 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:27.282805 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:27.282775 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:27.899766 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:27.899726 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:28.022982 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:28.022953 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:29.637013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:29.636981 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:29.695284 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:29.695246 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:30.121914 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:30.121810 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:30.321566 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:30.321529 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:35.979119 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:35.979083 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:36.080052 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:36.080011 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:38.492157 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:38.492124 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:38.593979 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:38.593943 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:38.697577 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:38.697539 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:40.721021 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:40.720990 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m"] Jul 09 18:32:40.724161 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:40.724142 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:40.733692 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:40.733655 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r6xjq\"" Jul 09 18:32:40.733849 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:40.733666 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 18:32:40.735407 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:40.735383 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 18:32:40.744757 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:40.744733 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m"] Jul 09 18:32:40.820549 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:40.820510 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wngt4\" (UniqueName: \"kubernetes.io/projected/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e-kube-api-access-wngt4\") pod \"kuadrant-operator-controller-manager-55467d9f7-cd69m\" (UID: \"8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:40.820729 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:40.820643 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-cd69m\" (UID: \"8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:40.921624 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:40.921591 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-cd69m\" (UID: \"8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:40.921794 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:40.921630 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wngt4\" (UniqueName: \"kubernetes.io/projected/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e-kube-api-access-wngt4\") pod \"kuadrant-operator-controller-manager-55467d9f7-cd69m\" (UID: \"8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:40.921984 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:40.921963 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-cd69m\" (UID: \"8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:40.939135 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:40.939109 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wngt4\" (UniqueName: \"kubernetes.io/projected/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e-kube-api-access-wngt4\") pod \"kuadrant-operator-controller-manager-55467d9f7-cd69m\" (UID: \"8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:41.038870 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.038794 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:41.144334 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.144295 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m"] Jul 09 18:32:41.150083 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.150057 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m"] Jul 09 18:32:41.171400 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.171374 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4"] Jul 09 18:32:41.174900 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.174878 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" Jul 09 18:32:41.192400 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.192372 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4"] Jul 09 18:32:41.267809 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:32:41.267762 2558 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 09 18:32:41.267809 ip-10-0-138-253 kubenswrapper[2558]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-cd69m_kuadrant-system_8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e_0(3328605c0a10c159725bce9271353c5b87b535961e4bed53d57f32ecf9013220): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-cd69m to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3328605c0a10c159725bce9271353c5b87b535961e4bed53d57f32ecf9013220" Netns:"/var/run/netns/b4b6269c-9401-4a8c-9b38-a1a00b530877" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-cd69m;K8S_POD_INFRA_CONTAINER_ID=3328605c0a10c159725bce9271353c5b87b535961e4bed53d57f32ecf9013220;K8S_POD_UID=8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m] networking: Multus: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod kuadrant-operator-controller-manager-55467d9f7-cd69m in out of cluster comm: SetNetworkStatus: failed to update the pod kuadrant-operator-controller-manager-55467d9f7-cd69m in out of cluster comm: status update failed for pod /: pods "kuadrant-operator-controller-manager-55467d9f7-cd69m" not found Jul 09 18:32:41.267809 ip-10-0-138-253 kubenswrapper[2558]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 09 18:32:41.267809 ip-10-0-138-253 kubenswrapper[2558]: > Jul 09 18:32:41.268057 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:32:41.267839 2558 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 09 18:32:41.268057 ip-10-0-138-253 kubenswrapper[2558]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-cd69m_kuadrant-system_8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e_0(3328605c0a10c159725bce9271353c5b87b535961e4bed53d57f32ecf9013220): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-cd69m to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3328605c0a10c159725bce9271353c5b87b535961e4bed53d57f32ecf9013220" Netns:"/var/run/netns/b4b6269c-9401-4a8c-9b38-a1a00b530877" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-cd69m;K8S_POD_INFRA_CONTAINER_ID=3328605c0a10c159725bce9271353c5b87b535961e4bed53d57f32ecf9013220;K8S_POD_UID=8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m] networking: Multus: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod kuadrant-operator-controller-manager-55467d9f7-cd69m in out of cluster comm: SetNetworkStatus: failed to update the pod kuadrant-operator-controller-manager-55467d9f7-cd69m in out of cluster comm: status update failed for pod /: pods "kuadrant-operator-controller-manager-55467d9f7-cd69m" not found Jul 09 18:32:41.268057 ip-10-0-138-253 kubenswrapper[2558]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 09 18:32:41.268057 ip-10-0-138-253 kubenswrapper[2558]: > pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:41.325365 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.325286 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-h7lx4\" (UID: \"6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" Jul 09 18:32:41.325507 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.325431 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6hvr\" (UniqueName: \"kubernetes.io/projected/6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce-kube-api-access-z6hvr\") pod \"kuadrant-operator-controller-manager-55467d9f7-h7lx4\" (UID: \"6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" Jul 09 18:32:41.426625 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.426568 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6hvr\" (UniqueName: \"kubernetes.io/projected/6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce-kube-api-access-z6hvr\") pod \"kuadrant-operator-controller-manager-55467d9f7-h7lx4\" (UID: \"6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" Jul 09 18:32:41.426851 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.426645 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-h7lx4\" (UID: \"6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" Jul 09 18:32:41.427072 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.427050 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-h7lx4\" (UID: \"6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" Jul 09 18:32:41.436149 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.436120 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6hvr\" (UniqueName: \"kubernetes.io/projected/6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce-kube-api-access-z6hvr\") pod \"kuadrant-operator-controller-manager-55467d9f7-h7lx4\" (UID: \"6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" Jul 09 18:32:41.486101 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.486076 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" Jul 09 18:32:41.625852 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.625829 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4"] Jul 09 18:32:41.628105 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:32:41.628067 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c15df7f_e13a_4a1f_8d6c_a9917c88c3ce.slice/crio-b23ed2b765ffd0dd574296dffbb75c77e2ad3fb7cd0221c2d39e274184250a19 WatchSource:0}: Error finding container b23ed2b765ffd0dd574296dffbb75c77e2ad3fb7cd0221c2d39e274184250a19: Status 404 returned error can't find the container with id b23ed2b765ffd0dd574296dffbb75c77e2ad3fb7cd0221c2d39e274184250a19 Jul 09 18:32:41.754802 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.754765 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:41.852965 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:41.852920 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:42.266218 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:42.266179 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" event={"ID":"6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce","Type":"ContainerStarted","Data":"b23ed2b765ffd0dd574296dffbb75c77e2ad3fb7cd0221c2d39e274184250a19"} Jul 09 18:32:42.266387 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:42.266297 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:42.274511 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:42.274487 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:42.277096 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:32:42.277039 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-cd69m\" is forbidden: User \"system:node:ip-10-0-138-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-253.ec2.internal' and this object" podUID="8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:42.333974 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:42.333940 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e-extensions-socket-volume\") pod \"8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e\" (UID: \"8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e\") " Jul 09 18:32:42.334140 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:42.334005 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wngt4\" (UniqueName: \"kubernetes.io/projected/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e-kube-api-access-wngt4\") pod \"8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e\" (UID: \"8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e\") " Jul 09 18:32:42.334276 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:42.334251 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e" (UID: "8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:32:42.335980 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:42.335952 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e-kube-api-access-wngt4" (OuterVolumeSpecName: "kube-api-access-wngt4") pod "8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e" (UID: "8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e"). InnerVolumeSpecName "kube-api-access-wngt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:32:42.369395 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:42.369354 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:42.435546 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:42.435516 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wngt4\" (UniqueName: \"kubernetes.io/projected/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e-kube-api-access-wngt4\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:32:42.435546 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:42.435544 2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e-extensions-socket-volume\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:32:42.475771 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:42.475164 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:43.271172 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:43.271138 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:43.273706 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:32:43.273665 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-cd69m\" is forbidden: User \"system:node:ip-10-0-138-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-253.ec2.internal' and this object" podUID="8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:43.281305 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:32:43.281274 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-cd69m\" is forbidden: User \"system:node:ip-10-0-138-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-253.ec2.internal' and this object" podUID="8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:43.559107 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:32:43.559030 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-cd69m\" is forbidden: User \"system:node:ip-10-0-138-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-253.ec2.internal' and this object" podUID="8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cd69m" Jul 09 18:32:43.559303 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:43.559282 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e" path="/var/lib/kubelet/pods/8aa2c161-5b3d-4899-8a08-7c09ae9b6d4e/volumes" Jul 09 18:32:46.328252 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:46.328212 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:46.429018 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:46.428986 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:47.291112 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:47.291068 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" event={"ID":"6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce","Type":"ContainerStarted","Data":"285518961a1c985a75c3afc0aabd761e9bc5f0a436474cc8ccf457625882b754"} Jul 09 18:32:47.291310 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:47.291231 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" Jul 09 18:32:47.338282 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:47.338235 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" podStartSLOduration=1.5029678340000001 podStartE2EDuration="6.338219914s" totalImagesPullingTime="4.83525208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:32:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:32:47.337132119 +0000 UTC m=+434.401797794" watchObservedRunningTime="2026-07-09 18:32:47.338219914 +0000 UTC m=+434.402885588" Jul 09 18:32:49.731098 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:49.731062 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:49.798094 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:49.798064 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:55.344764 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:55.344648 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:55.446872 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:55.446820 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:32:58.297137 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:32:58.297107 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" Jul 09 18:33:14.697732 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:14.697666 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:33:14.942904 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:14.942861 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:33:16.551421 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:16.551388 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4"] Jul 09 18:33:16.551973 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:16.551576 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" podUID="6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce" containerName="manager" containerID="cri-o://285518961a1c985a75c3afc0aabd761e9bc5f0a436474cc8ccf457625882b754" gracePeriod=10 Jul 09 18:33:16.789969 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:16.789945 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" Jul 09 18:33:16.814809 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:16.814740 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6hvr\" (UniqueName: \"kubernetes.io/projected/6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce-kube-api-access-z6hvr\") pod \"6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce\" (UID: \"6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce\") " Jul 09 18:33:16.814809 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:16.814808 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce-extensions-socket-volume\") pod \"6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce\" (UID: \"6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce\") " Jul 09 18:33:16.815387 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:16.815355 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce" (UID: "6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:33:16.815554 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:16.815539 2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce-extensions-socket-volume\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:33:16.817050 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:16.817021 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce-kube-api-access-z6hvr" (OuterVolumeSpecName: "kube-api-access-z6hvr") pod "6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce" (UID: "6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce"). InnerVolumeSpecName "kube-api-access-z6hvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:33:16.916572 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:16.916524 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6hvr\" (UniqueName: \"kubernetes.io/projected/6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce-kube-api-access-z6hvr\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:33:17.416514 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:17.416483 2558 generic.go:356] "Generic (PLEG): container finished" podID="6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce" containerID="285518961a1c985a75c3afc0aabd761e9bc5f0a436474cc8ccf457625882b754" exitCode=0 Jul 09 18:33:17.416688 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:17.416527 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" Jul 09 18:33:17.416688 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:17.416578 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" event={"ID":"6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce","Type":"ContainerDied","Data":"285518961a1c985a75c3afc0aabd761e9bc5f0a436474cc8ccf457625882b754"} Jul 09 18:33:17.416688 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:17.416625 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4" event={"ID":"6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce","Type":"ContainerDied","Data":"b23ed2b765ffd0dd574296dffbb75c77e2ad3fb7cd0221c2d39e274184250a19"} Jul 09 18:33:17.416688 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:17.416646 2558 scope.go:117] "RemoveContainer" containerID="285518961a1c985a75c3afc0aabd761e9bc5f0a436474cc8ccf457625882b754" Jul 09 18:33:17.425379 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:17.425361 2558 scope.go:117] "RemoveContainer" containerID="285518961a1c985a75c3afc0aabd761e9bc5f0a436474cc8ccf457625882b754" Jul 09 18:33:17.425653 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:33:17.425636 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285518961a1c985a75c3afc0aabd761e9bc5f0a436474cc8ccf457625882b754\": container with ID starting with 285518961a1c985a75c3afc0aabd761e9bc5f0a436474cc8ccf457625882b754 not found: ID does not exist" containerID="285518961a1c985a75c3afc0aabd761e9bc5f0a436474cc8ccf457625882b754" Jul 09 18:33:17.425750 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:17.425666 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285518961a1c985a75c3afc0aabd761e9bc5f0a436474cc8ccf457625882b754"} err="failed to get container status \"285518961a1c985a75c3afc0aabd761e9bc5f0a436474cc8ccf457625882b754\": rpc error: code = NotFound desc = could not find container \"285518961a1c985a75c3afc0aabd761e9bc5f0a436474cc8ccf457625882b754\": container with ID starting with 285518961a1c985a75c3afc0aabd761e9bc5f0a436474cc8ccf457625882b754 not found: ID does not exist" Jul 09 18:33:17.482446 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:17.482408 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4"] Jul 09 18:33:17.484813 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:17.484790 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-h7lx4"] Jul 09 18:33:17.558909 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:17.558878 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce" path="/var/lib/kubelet/pods/6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce/volumes" Jul 09 18:33:18.567150 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:18.567117 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:33:22.753845 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:22.753812 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:33:27.108955 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:27.108917 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:33:33.246637 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.246602 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:33:33.398524 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.398492 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:33:33.496052 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.496020 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:33:33.876479 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.876444 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:33:33.876759 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.876746 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce" containerName="manager" Jul 09 18:33:33.876810 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.876761 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce" containerName="manager" Jul 09 18:33:33.876844 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.876820 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c15df7f-e13a-4a1f-8d6c-a9917c88c3ce" containerName="manager" Jul 09 18:33:33.880792 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.880774 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-vr4w6" Jul 09 18:33:33.884799 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.884781 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-68mcm\"" Jul 09 18:33:33.884919 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.884846 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 18:33:33.885444 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.885429 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 18:33:33.885502 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.885479 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 18:33:33.890389 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.890368 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:33:33.922152 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.922119 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:33:33.952932 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.952906 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/babbc9ec-7b26-4818-bc05-dff504353120-config-file\") pod \"limitador-limitador-54d6d988d5-vr4w6\" (UID: \"babbc9ec-7b26-4818-bc05-dff504353120\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vr4w6" Jul 09 18:33:33.953038 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:33.953017 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k4zx\" (UniqueName: \"kubernetes.io/projected/babbc9ec-7b26-4818-bc05-dff504353120-kube-api-access-6k4zx\") pod \"limitador-limitador-54d6d988d5-vr4w6\" (UID: \"babbc9ec-7b26-4818-bc05-dff504353120\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vr4w6" Jul 09 18:33:34.054202 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.054164 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/babbc9ec-7b26-4818-bc05-dff504353120-config-file\") pod \"limitador-limitador-54d6d988d5-vr4w6\" (UID: \"babbc9ec-7b26-4818-bc05-dff504353120\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vr4w6" Jul 09 18:33:34.054362 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.054252 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k4zx\" (UniqueName: \"kubernetes.io/projected/babbc9ec-7b26-4818-bc05-dff504353120-kube-api-access-6k4zx\") pod \"limitador-limitador-54d6d988d5-vr4w6\" (UID: \"babbc9ec-7b26-4818-bc05-dff504353120\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vr4w6" Jul 09 18:33:34.054813 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.054794 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/babbc9ec-7b26-4818-bc05-dff504353120-config-file\") pod \"limitador-limitador-54d6d988d5-vr4w6\" (UID: \"babbc9ec-7b26-4818-bc05-dff504353120\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vr4w6" Jul 09 18:33:34.062499 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.062477 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k4zx\" (UniqueName: \"kubernetes.io/projected/babbc9ec-7b26-4818-bc05-dff504353120-kube-api-access-6k4zx\") pod \"limitador-limitador-54d6d988d5-vr4w6\" (UID: \"babbc9ec-7b26-4818-bc05-dff504353120\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vr4w6" Jul 09 18:33:34.113042 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.113013 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-tdlm8"] Jul 09 18:33:34.116228 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.116212 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tdlm8" Jul 09 18:33:34.118485 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.118459 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pd46m\"" Jul 09 18:33:34.124775 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.124756 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-tdlm8"] Jul 09 18:33:34.146928 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.146897 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:33:34.155359 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.155340 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f6fv\" (UniqueName: \"kubernetes.io/projected/65d66e9f-c476-4cc1-b8ff-81d0a29c9099-kube-api-access-4f6fv\") pod \"authorino-58cd955675-tdlm8\" (UID: \"65d66e9f-c476-4cc1-b8ff-81d0a29c9099\") " pod="kuadrant-system/authorino-58cd955675-tdlm8" Jul 09 18:33:34.191371 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.191339 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-vr4w6" Jul 09 18:33:34.208919 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.208894 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:33:34.255980 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.255945 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f6fv\" (UniqueName: \"kubernetes.io/projected/65d66e9f-c476-4cc1-b8ff-81d0a29c9099-kube-api-access-4f6fv\") pod \"authorino-58cd955675-tdlm8\" (UID: \"65d66e9f-c476-4cc1-b8ff-81d0a29c9099\") " pod="kuadrant-system/authorino-58cd955675-tdlm8" Jul 09 18:33:34.265064 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.265027 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f6fv\" (UniqueName: \"kubernetes.io/projected/65d66e9f-c476-4cc1-b8ff-81d0a29c9099-kube-api-access-4f6fv\") pod \"authorino-58cd955675-tdlm8\" (UID: \"65d66e9f-c476-4cc1-b8ff-81d0a29c9099\") " pod="kuadrant-system/authorino-58cd955675-tdlm8" Jul 09 18:33:34.322497 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.322468 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:33:34.326313 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:33:34.326286 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabbc9ec_7b26_4818_bc05_dff504353120.slice/crio-bedd6a4bcac68c87849625e0f73bf6ebe669a5292cc4483d9b8fe9f3d30bf046 WatchSource:0}: Error finding container bedd6a4bcac68c87849625e0f73bf6ebe669a5292cc4483d9b8fe9f3d30bf046: Status 404 returned error can't find the container with id bedd6a4bcac68c87849625e0f73bf6ebe669a5292cc4483d9b8fe9f3d30bf046 Jul 09 18:33:34.384964 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.384937 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-rrr7p"] Jul 09 18:33:34.389075 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.389058 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-rrr7p" Jul 09 18:33:34.395916 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.395888 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rrr7p"] Jul 09 18:33:34.425854 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.425783 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tdlm8" Jul 09 18:33:34.457579 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.457547 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz76j\" (UniqueName: \"kubernetes.io/projected/7a8beed4-1bc6-46b5-afb5-02291f343f0d-kube-api-access-qz76j\") pod \"authorino-75d556bd77-rrr7p\" (UID: \"7a8beed4-1bc6-46b5-afb5-02291f343f0d\") " pod="kuadrant-system/authorino-75d556bd77-rrr7p" Jul 09 18:33:34.487584 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.487556 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-vr4w6" event={"ID":"babbc9ec-7b26-4818-bc05-dff504353120","Type":"ContainerStarted","Data":"bedd6a4bcac68c87849625e0f73bf6ebe669a5292cc4483d9b8fe9f3d30bf046"} Jul 09 18:33:34.553687 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.553656 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-tdlm8"] Jul 09 18:33:34.556619 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:33:34.556594 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65d66e9f_c476_4cc1_b8ff_81d0a29c9099.slice/crio-ff8318f3e16183ea2330f0a1d3d76e932cd18f5910d871b8bda22c3c2a830121 WatchSource:0}: Error finding container ff8318f3e16183ea2330f0a1d3d76e932cd18f5910d871b8bda22c3c2a830121: Status 404 returned error can't find the container with id ff8318f3e16183ea2330f0a1d3d76e932cd18f5910d871b8bda22c3c2a830121 Jul 09 18:33:34.557993 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.557973 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz76j\" (UniqueName: \"kubernetes.io/projected/7a8beed4-1bc6-46b5-afb5-02291f343f0d-kube-api-access-qz76j\") pod \"authorino-75d556bd77-rrr7p\" (UID: \"7a8beed4-1bc6-46b5-afb5-02291f343f0d\") " pod="kuadrant-system/authorino-75d556bd77-rrr7p" Jul 09 18:33:34.570101 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.570075 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz76j\" (UniqueName: \"kubernetes.io/projected/7a8beed4-1bc6-46b5-afb5-02291f343f0d-kube-api-access-qz76j\") pod \"authorino-75d556bd77-rrr7p\" (UID: \"7a8beed4-1bc6-46b5-afb5-02291f343f0d\") " pod="kuadrant-system/authorino-75d556bd77-rrr7p" Jul 09 18:33:34.698890 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.698808 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-rrr7p" Jul 09 18:33:34.838666 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:34.838640 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rrr7p"] Jul 09 18:33:34.840876 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:33:34.840848 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a8beed4_1bc6_46b5_afb5_02291f343f0d.slice/crio-acfc502af3009591f40796ffae335bfba16f031d158f7c4926acecc1bda9ecda WatchSource:0}: Error finding container acfc502af3009591f40796ffae335bfba16f031d158f7c4926acecc1bda9ecda: Status 404 returned error can't find the container with id acfc502af3009591f40796ffae335bfba16f031d158f7c4926acecc1bda9ecda Jul 09 18:33:35.501435 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:35.501391 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tdlm8" event={"ID":"65d66e9f-c476-4cc1-b8ff-81d0a29c9099","Type":"ContainerStarted","Data":"ff8318f3e16183ea2330f0a1d3d76e932cd18f5910d871b8bda22c3c2a830121"} Jul 09 18:33:35.503310 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:35.503280 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-rrr7p" event={"ID":"7a8beed4-1bc6-46b5-afb5-02291f343f0d","Type":"ContainerStarted","Data":"acfc502af3009591f40796ffae335bfba16f031d158f7c4926acecc1bda9ecda"} Jul 09 18:33:40.538362 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:40.538319 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-vr4w6" event={"ID":"babbc9ec-7b26-4818-bc05-dff504353120","Type":"ContainerStarted","Data":"e8cbb59c691e9812ce94604ea41db600c3ceeb2616bf9cec1e276f5be4273899"} Jul 09 18:33:40.538826 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:40.538482 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-vr4w6" Jul 09 18:33:40.539831 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:40.539806 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tdlm8" event={"ID":"65d66e9f-c476-4cc1-b8ff-81d0a29c9099","Type":"ContainerStarted","Data":"c30139b5839dbc4cdaa775cba8efda42639dd7c37162484b144d1732fbc860b2"} Jul 09 18:33:40.541224 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:40.541201 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-rrr7p" event={"ID":"7a8beed4-1bc6-46b5-afb5-02291f343f0d","Type":"ContainerStarted","Data":"0429a60e83293b40a2e88f7e934bb5c303fe2c2939400768b23fd0fd479b8619"} Jul 09 18:33:40.558108 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:40.558052 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-vr4w6" podStartSLOduration=2.424088568 podStartE2EDuration="7.558036443s" totalImagesPullingTime="5.133947875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:33:40.557731669 +0000 UTC m=+487.622397340" watchObservedRunningTime="2026-07-09 18:33:40.558036443 +0000 UTC m=+487.622702115" Jul 09 18:33:40.574481 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:40.574434 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-rrr7p" podStartSLOduration=1.962383141 podStartE2EDuration="6.574421145s" totalImagesPullingTime="4.612038004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:33:40.57356767 +0000 UTC m=+487.638233347" watchObservedRunningTime="2026-07-09 18:33:40.574421145 +0000 UTC m=+487.639086862" Jul 09 18:33:40.590072 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:40.590023 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-tdlm8" podStartSLOduration=1.7323159719999999 podStartE2EDuration="6.59000507s" totalImagesPullingTime="4.857689098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:33:40.588920137 +0000 UTC m=+487.653585811" watchObservedRunningTime="2026-07-09 18:33:40.59000507 +0000 UTC m=+487.654670744" Jul 09 18:33:40.618848 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:40.618813 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-tdlm8"] Jul 09 18:33:42.549873 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:42.549836 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-tdlm8" podUID="65d66e9f-c476-4cc1-b8ff-81d0a29c9099" containerName="authorino" containerID="cri-o://c30139b5839dbc4cdaa775cba8efda42639dd7c37162484b144d1732fbc860b2" gracePeriod=30 Jul 09 18:33:42.802805 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:42.802738 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tdlm8" Jul 09 18:33:42.937062 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:42.937022 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f6fv\" (UniqueName: \"kubernetes.io/projected/65d66e9f-c476-4cc1-b8ff-81d0a29c9099-kube-api-access-4f6fv\") pod \"65d66e9f-c476-4cc1-b8ff-81d0a29c9099\" (UID: \"65d66e9f-c476-4cc1-b8ff-81d0a29c9099\") " Jul 09 18:33:42.939105 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:42.939081 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d66e9f-c476-4cc1-b8ff-81d0a29c9099-kube-api-access-4f6fv" (OuterVolumeSpecName: "kube-api-access-4f6fv") pod "65d66e9f-c476-4cc1-b8ff-81d0a29c9099" (UID: "65d66e9f-c476-4cc1-b8ff-81d0a29c9099"). InnerVolumeSpecName "kube-api-access-4f6fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:33:43.037900 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:43.037869 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f6fv\" (UniqueName: \"kubernetes.io/projected/65d66e9f-c476-4cc1-b8ff-81d0a29c9099-kube-api-access-4f6fv\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:33:43.555790 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:43.555753 2558 generic.go:356] "Generic (PLEG): container finished" podID="65d66e9f-c476-4cc1-b8ff-81d0a29c9099" containerID="c30139b5839dbc4cdaa775cba8efda42639dd7c37162484b144d1732fbc860b2" exitCode=0 Jul 09 18:33:43.556253 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:43.555836 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tdlm8" Jul 09 18:33:43.558436 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:43.558408 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tdlm8" event={"ID":"65d66e9f-c476-4cc1-b8ff-81d0a29c9099","Type":"ContainerDied","Data":"c30139b5839dbc4cdaa775cba8efda42639dd7c37162484b144d1732fbc860b2"} Jul 09 18:33:43.558436 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:43.558441 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tdlm8" event={"ID":"65d66e9f-c476-4cc1-b8ff-81d0a29c9099","Type":"ContainerDied","Data":"ff8318f3e16183ea2330f0a1d3d76e932cd18f5910d871b8bda22c3c2a830121"} Jul 09 18:33:43.558620 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:43.558459 2558 scope.go:117] "RemoveContainer" containerID="c30139b5839dbc4cdaa775cba8efda42639dd7c37162484b144d1732fbc860b2" Jul 09 18:33:43.567558 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:43.567542 2558 scope.go:117] "RemoveContainer" containerID="c30139b5839dbc4cdaa775cba8efda42639dd7c37162484b144d1732fbc860b2" Jul 09 18:33:43.567919 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:33:43.567899 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30139b5839dbc4cdaa775cba8efda42639dd7c37162484b144d1732fbc860b2\": container with ID starting with c30139b5839dbc4cdaa775cba8efda42639dd7c37162484b144d1732fbc860b2 not found: ID does not exist" containerID="c30139b5839dbc4cdaa775cba8efda42639dd7c37162484b144d1732fbc860b2" Jul 09 18:33:43.567973 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:43.567926 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30139b5839dbc4cdaa775cba8efda42639dd7c37162484b144d1732fbc860b2"} err="failed to get container status \"c30139b5839dbc4cdaa775cba8efda42639dd7c37162484b144d1732fbc860b2\": rpc error: code = NotFound desc = could not find container \"c30139b5839dbc4cdaa775cba8efda42639dd7c37162484b144d1732fbc860b2\": container with ID starting with c30139b5839dbc4cdaa775cba8efda42639dd7c37162484b144d1732fbc860b2 not found: ID does not exist" Jul 09 18:33:43.579561 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:43.579539 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-tdlm8"] Jul 09 18:33:43.583280 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:43.583254 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-tdlm8"] Jul 09 18:33:45.558870 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:45.558840 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65d66e9f-c476-4cc1-b8ff-81d0a29c9099" path="/var/lib/kubelet/pods/65d66e9f-c476-4cc1-b8ff-81d0a29c9099/volumes" Jul 09 18:33:51.546862 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:51.546830 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-vr4w6" Jul 09 18:33:56.132412 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:56.132365 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:33:56.231132 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:56.231099 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:33:56.615803 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:33:56.615726 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:34:09.086000 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.085957 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:34:09.333118 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.333081 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9chlr"] Jul 09 18:34:09.333426 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.333413 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d66e9f-c476-4cc1-b8ff-81d0a29c9099" containerName="authorino" Jul 09 18:34:09.333467 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.333427 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d66e9f-c476-4cc1-b8ff-81d0a29c9099" containerName="authorino" Jul 09 18:34:09.333500 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.333488 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="65d66e9f-c476-4cc1-b8ff-81d0a29c9099" containerName="authorino" Jul 09 18:34:09.341058 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.341007 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9chlr" Jul 09 18:34:09.344667 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.344637 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9chlr"] Jul 09 18:34:09.355123 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.355098 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9sv\" (UniqueName: \"kubernetes.io/projected/6bd7df1e-aa21-4067-9221-d3ad1a1373d3-kube-api-access-2z9sv\") pod \"authorino-5d4b67c446-9chlr\" (UID: \"6bd7df1e-aa21-4067-9221-d3ad1a1373d3\") " pod="kuadrant-system/authorino-5d4b67c446-9chlr" Jul 09 18:34:09.456053 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.456020 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9sv\" (UniqueName: \"kubernetes.io/projected/6bd7df1e-aa21-4067-9221-d3ad1a1373d3-kube-api-access-2z9sv\") pod \"authorino-5d4b67c446-9chlr\" (UID: \"6bd7df1e-aa21-4067-9221-d3ad1a1373d3\") " pod="kuadrant-system/authorino-5d4b67c446-9chlr" Jul 09 18:34:09.465344 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.465317 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9sv\" (UniqueName: \"kubernetes.io/projected/6bd7df1e-aa21-4067-9221-d3ad1a1373d3-kube-api-access-2z9sv\") pod \"authorino-5d4b67c446-9chlr\" (UID: \"6bd7df1e-aa21-4067-9221-d3ad1a1373d3\") " pod="kuadrant-system/authorino-5d4b67c446-9chlr" Jul 09 18:34:09.559021 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.558989 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9chlr"] Jul 09 18:34:09.559201 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.559169 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9chlr" Jul 09 18:34:09.583912 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.583882 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fc4b46c74-vkv97"] Jul 09 18:34:09.588174 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.588154 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc4b46c74-vkv97" Jul 09 18:34:09.596022 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.595990 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fc4b46c74-vkv97"] Jul 09 18:34:09.657450 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.657401 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spr8j\" (UniqueName: \"kubernetes.io/projected/dc9315d6-6ec9-40ee-b686-5ee41820e21e-kube-api-access-spr8j\") pod \"authorino-5fc4b46c74-vkv97\" (UID: \"dc9315d6-6ec9-40ee-b686-5ee41820e21e\") " pod="kuadrant-system/authorino-5fc4b46c74-vkv97" Jul 09 18:34:09.695712 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.695677 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9chlr"] Jul 09 18:34:09.698252 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:34:09.698226 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bd7df1e_aa21_4067_9221_d3ad1a1373d3.slice/crio-ecb44731162a05916a308beda8d3af8c97997015c227b1d82941eaddddb2df25 WatchSource:0}: Error finding container ecb44731162a05916a308beda8d3af8c97997015c227b1d82941eaddddb2df25: Status 404 returned error can't find the container with id ecb44731162a05916a308beda8d3af8c97997015c227b1d82941eaddddb2df25 Jul 09 18:34:09.758905 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.758872 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spr8j\" (UniqueName: \"kubernetes.io/projected/dc9315d6-6ec9-40ee-b686-5ee41820e21e-kube-api-access-spr8j\") pod \"authorino-5fc4b46c74-vkv97\" (UID: \"dc9315d6-6ec9-40ee-b686-5ee41820e21e\") " pod="kuadrant-system/authorino-5fc4b46c74-vkv97" Jul 09 18:34:09.768067 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.768039 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spr8j\" (UniqueName: \"kubernetes.io/projected/dc9315d6-6ec9-40ee-b686-5ee41820e21e-kube-api-access-spr8j\") pod \"authorino-5fc4b46c74-vkv97\" (UID: \"dc9315d6-6ec9-40ee-b686-5ee41820e21e\") " pod="kuadrant-system/authorino-5fc4b46c74-vkv97" Jul 09 18:34:09.898428 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.898403 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc4b46c74-vkv97" Jul 09 18:34:09.900470 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:09.900447 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:34:10.025334 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.025294 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fc4b46c74-vkv97"] Jul 09 18:34:10.027990 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:34:10.027967 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc9315d6_6ec9_40ee_b686_5ee41820e21e.slice/crio-110506242b7a1078449c7cd82550f1a147a8af164415b3efd5609c3930e35301 WatchSource:0}: Error finding container 110506242b7a1078449c7cd82550f1a147a8af164415b3efd5609c3930e35301: Status 404 returned error can't find the container with id 110506242b7a1078449c7cd82550f1a147a8af164415b3efd5609c3930e35301 Jul 09 18:34:10.115779 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.115742 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fc4b46c74-vkv97"] Jul 09 18:34:10.147491 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.147458 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77c849fddb-gkdt9"] Jul 09 18:34:10.150361 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.150319 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77c849fddb-gkdt9" Jul 09 18:34:10.152849 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.152831 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 09 18:34:10.160218 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.160190 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77c849fddb-gkdt9"] Jul 09 18:34:10.162854 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.162833 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0-tls-cert\") pod \"authorino-77c849fddb-gkdt9\" (UID: \"59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0\") " pod="kuadrant-system/authorino-77c849fddb-gkdt9" Jul 09 18:34:10.162959 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.162859 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nb4x\" (UniqueName: \"kubernetes.io/projected/59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0-kube-api-access-9nb4x\") pod \"authorino-77c849fddb-gkdt9\" (UID: \"59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0\") " pod="kuadrant-system/authorino-77c849fddb-gkdt9" Jul 09 18:34:10.264313 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.264267 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0-tls-cert\") pod \"authorino-77c849fddb-gkdt9\" (UID: \"59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0\") " pod="kuadrant-system/authorino-77c849fddb-gkdt9" Jul 09 18:34:10.264583 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.264322 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nb4x\" (UniqueName: \"kubernetes.io/projected/59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0-kube-api-access-9nb4x\") pod \"authorino-77c849fddb-gkdt9\" (UID: \"59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0\") " pod="kuadrant-system/authorino-77c849fddb-gkdt9" Jul 09 18:34:10.267109 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.267086 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0-tls-cert\") pod \"authorino-77c849fddb-gkdt9\" (UID: \"59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0\") " pod="kuadrant-system/authorino-77c849fddb-gkdt9" Jul 09 18:34:10.276012 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.275991 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nb4x\" (UniqueName: \"kubernetes.io/projected/59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0-kube-api-access-9nb4x\") pod \"authorino-77c849fddb-gkdt9\" (UID: \"59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0\") " pod="kuadrant-system/authorino-77c849fddb-gkdt9" Jul 09 18:34:10.460880 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.460797 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77c849fddb-gkdt9" Jul 09 18:34:10.593207 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.593178 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77c849fddb-gkdt9"] Jul 09 18:34:10.596568 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:34:10.596542 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59473d5a_f14e_4d6f_aa62_2bfbb01fb3a0.slice/crio-e5fdfe30aa02264803416712cab13f6149f923acdb8eb78ce7caa8d1c77337f0 WatchSource:0}: Error finding container e5fdfe30aa02264803416712cab13f6149f923acdb8eb78ce7caa8d1c77337f0: Status 404 returned error can't find the container with id e5fdfe30aa02264803416712cab13f6149f923acdb8eb78ce7caa8d1c77337f0 Jul 09 18:34:10.671491 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.671454 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77c849fddb-gkdt9" event={"ID":"59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0","Type":"ContainerStarted","Data":"e5fdfe30aa02264803416712cab13f6149f923acdb8eb78ce7caa8d1c77337f0"} Jul 09 18:34:10.673098 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.673039 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5fc4b46c74-vkv97" podUID="dc9315d6-6ec9-40ee-b686-5ee41820e21e" containerName="authorino" containerID="cri-o://477972692cd89dfee57835f3e8bd66cbeda050705e6f466ec745505c55c51149" gracePeriod=30 Jul 09 18:34:10.673243 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.673044 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc4b46c74-vkv97" event={"ID":"dc9315d6-6ec9-40ee-b686-5ee41820e21e","Type":"ContainerStarted","Data":"477972692cd89dfee57835f3e8bd66cbeda050705e6f466ec745505c55c51149"} Jul 09 18:34:10.673243 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.673135 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc4b46c74-vkv97" event={"ID":"dc9315d6-6ec9-40ee-b686-5ee41820e21e","Type":"ContainerStarted","Data":"110506242b7a1078449c7cd82550f1a147a8af164415b3efd5609c3930e35301"} Jul 09 18:34:10.674481 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.674458 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9chlr" event={"ID":"6bd7df1e-aa21-4067-9221-d3ad1a1373d3","Type":"ContainerStarted","Data":"d2dcc1956e89485d9718247483c160f0be75e330aea35f6b399d739f2335fd41"} Jul 09 18:34:10.674578 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.674489 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9chlr" event={"ID":"6bd7df1e-aa21-4067-9221-d3ad1a1373d3","Type":"ContainerStarted","Data":"ecb44731162a05916a308beda8d3af8c97997015c227b1d82941eaddddb2df25"} Jul 09 18:34:10.674578 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.674553 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-9chlr" podUID="6bd7df1e-aa21-4067-9221-d3ad1a1373d3" containerName="authorino" containerID="cri-o://d2dcc1956e89485d9718247483c160f0be75e330aea35f6b399d739f2335fd41" gracePeriod=30 Jul 09 18:34:10.689127 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.689086 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5fc4b46c74-vkv97" podStartSLOduration=1.3775338320000001 podStartE2EDuration="1.689072675s" totalImagesPullingTime="311.538843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:34:10.688509268 +0000 UTC m=+517.753174943" watchObservedRunningTime="2026-07-09 18:34:10.689072675 +0000 UTC m=+517.753738349" Jul 09 18:34:10.704919 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.704870 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-9chlr" podStartSLOduration=1.251123067 podStartE2EDuration="1.704856945s" totalImagesPullingTime="453.733878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:34:10.70354392 +0000 UTC m=+517.768209595" watchObservedRunningTime="2026-07-09 18:34:10.704856945 +0000 UTC m=+517.769522618" Jul 09 18:34:10.923246 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.923221 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc4b46c74-vkv97" Jul 09 18:34:10.930263 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.930239 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9chlr" Jul 09 18:34:10.968427 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.968391 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z9sv\" (UniqueName: \"kubernetes.io/projected/6bd7df1e-aa21-4067-9221-d3ad1a1373d3-kube-api-access-2z9sv\") pod \"6bd7df1e-aa21-4067-9221-d3ad1a1373d3\" (UID: \"6bd7df1e-aa21-4067-9221-d3ad1a1373d3\") " Jul 09 18:34:10.968579 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.968448 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spr8j\" (UniqueName: \"kubernetes.io/projected/dc9315d6-6ec9-40ee-b686-5ee41820e21e-kube-api-access-spr8j\") pod \"dc9315d6-6ec9-40ee-b686-5ee41820e21e\" (UID: \"dc9315d6-6ec9-40ee-b686-5ee41820e21e\") " Jul 09 18:34:10.970508 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.970480 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9315d6-6ec9-40ee-b686-5ee41820e21e-kube-api-access-spr8j" (OuterVolumeSpecName: "kube-api-access-spr8j") pod "dc9315d6-6ec9-40ee-b686-5ee41820e21e" (UID: "dc9315d6-6ec9-40ee-b686-5ee41820e21e"). InnerVolumeSpecName "kube-api-access-spr8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:34:10.970508 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:10.970483 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd7df1e-aa21-4067-9221-d3ad1a1373d3-kube-api-access-2z9sv" (OuterVolumeSpecName: "kube-api-access-2z9sv") pod "6bd7df1e-aa21-4067-9221-d3ad1a1373d3" (UID: "6bd7df1e-aa21-4067-9221-d3ad1a1373d3"). InnerVolumeSpecName "kube-api-access-2z9sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:34:11.059911 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:34:11.059826 2558 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" image="quay.io/kuadrant/authorino:v0.24.0" Jul 09 18:34:11.060043 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:34:11.059938 2558 kuberuntime_manager.go:1459] "Unhandled Error" err="container authorino start failed in pod authorino-77c849fddb-gkdt9_kuadrant-system(59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" logger="UnhandledError" Jul 09 18:34:11.060043 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:34:11.059965 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"authorino\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="kuadrant-system/authorino-77c849fddb-gkdt9" podUID="59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0" Jul 09 18:34:11.069334 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.069307 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z9sv\" (UniqueName: \"kubernetes.io/projected/6bd7df1e-aa21-4067-9221-d3ad1a1373d3-kube-api-access-2z9sv\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:34:11.069334 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.069337 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spr8j\" (UniqueName: \"kubernetes.io/projected/dc9315d6-6ec9-40ee-b686-5ee41820e21e-kube-api-access-spr8j\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:34:11.685241 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.685202 2558 generic.go:356] "Generic (PLEG): container finished" podID="6bd7df1e-aa21-4067-9221-d3ad1a1373d3" containerID="d2dcc1956e89485d9718247483c160f0be75e330aea35f6b399d739f2335fd41" exitCode=0 Jul 09 18:34:11.685678 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.685268 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9chlr" Jul 09 18:34:11.685678 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.685298 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9chlr" event={"ID":"6bd7df1e-aa21-4067-9221-d3ad1a1373d3","Type":"ContainerDied","Data":"d2dcc1956e89485d9718247483c160f0be75e330aea35f6b399d739f2335fd41"} Jul 09 18:34:11.685678 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.685353 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9chlr" event={"ID":"6bd7df1e-aa21-4067-9221-d3ad1a1373d3","Type":"ContainerDied","Data":"ecb44731162a05916a308beda8d3af8c97997015c227b1d82941eaddddb2df25"} Jul 09 18:34:11.685678 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.685379 2558 scope.go:117] "RemoveContainer" containerID="d2dcc1956e89485d9718247483c160f0be75e330aea35f6b399d739f2335fd41" Jul 09 18:34:11.687172 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.686815 2558 generic.go:356] "Generic (PLEG): container finished" podID="dc9315d6-6ec9-40ee-b686-5ee41820e21e" containerID="477972692cd89dfee57835f3e8bd66cbeda050705e6f466ec745505c55c51149" exitCode=0 Jul 09 18:34:11.687172 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.686869 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc4b46c74-vkv97" Jul 09 18:34:11.687172 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.686908 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc4b46c74-vkv97" event={"ID":"dc9315d6-6ec9-40ee-b686-5ee41820e21e","Type":"ContainerDied","Data":"477972692cd89dfee57835f3e8bd66cbeda050705e6f466ec745505c55c51149"} Jul 09 18:34:11.687172 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.686948 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc4b46c74-vkv97" event={"ID":"dc9315d6-6ec9-40ee-b686-5ee41820e21e","Type":"ContainerDied","Data":"110506242b7a1078449c7cd82550f1a147a8af164415b3efd5609c3930e35301"} Jul 09 18:34:11.687655 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:34:11.687625 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"authorino\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/kuadrant/authorino:v0.24.0\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="kuadrant-system/authorino-77c849fddb-gkdt9" podUID="59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0" Jul 09 18:34:11.694215 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.694198 2558 scope.go:117] "RemoveContainer" containerID="d2dcc1956e89485d9718247483c160f0be75e330aea35f6b399d739f2335fd41" Jul 09 18:34:11.694456 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:34:11.694439 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2dcc1956e89485d9718247483c160f0be75e330aea35f6b399d739f2335fd41\": container with ID starting with d2dcc1956e89485d9718247483c160f0be75e330aea35f6b399d739f2335fd41 not found: ID does not exist" containerID="d2dcc1956e89485d9718247483c160f0be75e330aea35f6b399d739f2335fd41" Jul 09 18:34:11.694521 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.694469 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2dcc1956e89485d9718247483c160f0be75e330aea35f6b399d739f2335fd41"} err="failed to get container status \"d2dcc1956e89485d9718247483c160f0be75e330aea35f6b399d739f2335fd41\": rpc error: code = NotFound desc = could not find container \"d2dcc1956e89485d9718247483c160f0be75e330aea35f6b399d739f2335fd41\": container with ID starting with d2dcc1956e89485d9718247483c160f0be75e330aea35f6b399d739f2335fd41 not found: ID does not exist" Jul 09 18:34:11.694521 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.694494 2558 scope.go:117] "RemoveContainer" containerID="477972692cd89dfee57835f3e8bd66cbeda050705e6f466ec745505c55c51149" Jul 09 18:34:11.701677 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.701660 2558 scope.go:117] "RemoveContainer" containerID="477972692cd89dfee57835f3e8bd66cbeda050705e6f466ec745505c55c51149" Jul 09 18:34:11.701960 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:34:11.701943 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"477972692cd89dfee57835f3e8bd66cbeda050705e6f466ec745505c55c51149\": container with ID starting with 477972692cd89dfee57835f3e8bd66cbeda050705e6f466ec745505c55c51149 not found: ID does not exist" containerID="477972692cd89dfee57835f3e8bd66cbeda050705e6f466ec745505c55c51149" Jul 09 18:34:11.702011 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.701967 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477972692cd89dfee57835f3e8bd66cbeda050705e6f466ec745505c55c51149"} err="failed to get container status \"477972692cd89dfee57835f3e8bd66cbeda050705e6f466ec745505c55c51149\": rpc error: code = NotFound desc = could not find container \"477972692cd89dfee57835f3e8bd66cbeda050705e6f466ec745505c55c51149\": container with ID starting with 477972692cd89dfee57835f3e8bd66cbeda050705e6f466ec745505c55c51149 not found: ID does not exist" Jul 09 18:34:11.712528 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.712506 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fc4b46c74-vkv97"] Jul 09 18:34:11.715377 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.715358 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5fc4b46c74-vkv97"] Jul 09 18:34:11.732601 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.732549 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9chlr"] Jul 09 18:34:11.736000 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:11.735983 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9chlr"] Jul 09 18:34:13.559165 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:13.559135 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bd7df1e-aa21-4067-9221-d3ad1a1373d3" path="/var/lib/kubelet/pods/6bd7df1e-aa21-4067-9221-d3ad1a1373d3/volumes" Jul 09 18:34:13.559539 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:13.559443 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc9315d6-6ec9-40ee-b686-5ee41820e21e" path="/var/lib/kubelet/pods/dc9315d6-6ec9-40ee-b686-5ee41820e21e/volumes" Jul 09 18:34:25.753958 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:25.753921 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77c849fddb-gkdt9" event={"ID":"59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0","Type":"ContainerStarted","Data":"3db30557017e9b5779dff71e84bc9826fee503874244350eafc3bbbc748a77b9"} Jul 09 18:34:25.776844 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:25.776793 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-77c849fddb-gkdt9" podStartSLOduration=1.214719205 podStartE2EDuration="15.776777188s" totalImagesPullingTime="14.562057983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-09 18:34:25.775558148 +0000 UTC m=+532.840223832" watchObservedRunningTime="2026-07-09 18:34:25.776777188 +0000 UTC m=+532.841442862" Jul 09 18:34:25.812196 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:25.812164 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rrr7p"] Jul 09 18:34:25.812349 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:25.812332 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-rrr7p" podUID="7a8beed4-1bc6-46b5-afb5-02291f343f0d" containerName="authorino" containerID="cri-o://0429a60e83293b40a2e88f7e934bb5c303fe2c2939400768b23fd0fd479b8619" gracePeriod=30 Jul 09 18:34:26.052035 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:26.052012 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-rrr7p" Jul 09 18:34:26.096531 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:26.096505 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz76j\" (UniqueName: \"kubernetes.io/projected/7a8beed4-1bc6-46b5-afb5-02291f343f0d-kube-api-access-qz76j\") pod \"7a8beed4-1bc6-46b5-afb5-02291f343f0d\" (UID: \"7a8beed4-1bc6-46b5-afb5-02291f343f0d\") " Jul 09 18:34:26.098490 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:26.098469 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8beed4-1bc6-46b5-afb5-02291f343f0d-kube-api-access-qz76j" (OuterVolumeSpecName: "kube-api-access-qz76j") pod "7a8beed4-1bc6-46b5-afb5-02291f343f0d" (UID: "7a8beed4-1bc6-46b5-afb5-02291f343f0d"). InnerVolumeSpecName "kube-api-access-qz76j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:34:26.197344 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:26.197314 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz76j\" (UniqueName: \"kubernetes.io/projected/7a8beed4-1bc6-46b5-afb5-02291f343f0d-kube-api-access-qz76j\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:34:26.758997 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:26.758960 2558 generic.go:356] "Generic (PLEG): container finished" podID="7a8beed4-1bc6-46b5-afb5-02291f343f0d" containerID="0429a60e83293b40a2e88f7e934bb5c303fe2c2939400768b23fd0fd479b8619" exitCode=0 Jul 09 18:34:26.758997 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:26.759002 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-rrr7p" Jul 09 18:34:26.759441 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:26.759044 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-rrr7p" event={"ID":"7a8beed4-1bc6-46b5-afb5-02291f343f0d","Type":"ContainerDied","Data":"0429a60e83293b40a2e88f7e934bb5c303fe2c2939400768b23fd0fd479b8619"} Jul 09 18:34:26.759441 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:26.759086 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-rrr7p" event={"ID":"7a8beed4-1bc6-46b5-afb5-02291f343f0d","Type":"ContainerDied","Data":"acfc502af3009591f40796ffae335bfba16f031d158f7c4926acecc1bda9ecda"} Jul 09 18:34:26.759441 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:26.759102 2558 scope.go:117] "RemoveContainer" containerID="0429a60e83293b40a2e88f7e934bb5c303fe2c2939400768b23fd0fd479b8619" Jul 09 18:34:26.767808 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:26.767784 2558 scope.go:117] "RemoveContainer" containerID="0429a60e83293b40a2e88f7e934bb5c303fe2c2939400768b23fd0fd479b8619" Jul 09 18:34:26.768080 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:34:26.768062 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0429a60e83293b40a2e88f7e934bb5c303fe2c2939400768b23fd0fd479b8619\": container with ID starting with 0429a60e83293b40a2e88f7e934bb5c303fe2c2939400768b23fd0fd479b8619 not found: ID does not exist" containerID="0429a60e83293b40a2e88f7e934bb5c303fe2c2939400768b23fd0fd479b8619" Jul 09 18:34:26.768140 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:26.768088 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0429a60e83293b40a2e88f7e934bb5c303fe2c2939400768b23fd0fd479b8619"} err="failed to get container status \"0429a60e83293b40a2e88f7e934bb5c303fe2c2939400768b23fd0fd479b8619\": rpc error: code = NotFound desc = could not find container \"0429a60e83293b40a2e88f7e934bb5c303fe2c2939400768b23fd0fd479b8619\": container with ID starting with 0429a60e83293b40a2e88f7e934bb5c303fe2c2939400768b23fd0fd479b8619 not found: ID does not exist" Jul 09 18:34:26.780641 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:26.780618 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rrr7p"] Jul 09 18:34:26.782881 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:26.782861 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rrr7p"] Jul 09 18:34:27.558839 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:27.558810 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a8beed4-1bc6-46b5-afb5-02291f343f0d" path="/var/lib/kubelet/pods/7a8beed4-1bc6-46b5-afb5-02291f343f0d/volumes" Jul 09 18:34:30.584277 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:30.584247 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:34:30.665042 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:30.665000 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:34:30.721012 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:30.720979 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:34:30.929229 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:30.929196 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:34:34.003191 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:34.003156 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:34:34.105011 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:34.104978 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:34:47.280140 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:47.280106 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:34:47.331470 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:47.331443 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:34:48.600587 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:48.600553 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:34:48.701489 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:34:48.701460 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:35:06.451688 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:06.451650 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:35:06.855154 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:06.855075 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:35:07.039139 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:07.039107 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:35:07.447110 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:07.447073 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:35:07.544258 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:07.544219 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:35:07.677268 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:07.677236 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:35:17.501814 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:17.501784 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:35:17.590971 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:17.590939 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:35:30.490237 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:30.490195 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:35:32.824825 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:32.824792 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:35:33.108716 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:33.108605 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:35:33.316748 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:33.316720 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:35:33.317167 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:33.317149 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:35:55.343336 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:55.343250 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:35:56.188218 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:35:56.188177 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:36:02.779296 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:02.779263 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:36:04.093074 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:04.093038 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:36:08.525773 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:08.525688 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:36:10.299204 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:10.299168 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:36:16.089379 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.089347 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs"] Jul 09 18:36:16.089741 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.089640 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc9315d6-6ec9-40ee-b686-5ee41820e21e" containerName="authorino" Jul 09 18:36:16.089741 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.089650 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc9315d6-6ec9-40ee-b686-5ee41820e21e" containerName="authorino" Jul 09 18:36:16.089741 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.089667 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd7df1e-aa21-4067-9221-d3ad1a1373d3" containerName="authorino" Jul 09 18:36:16.089741 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.089673 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd7df1e-aa21-4067-9221-d3ad1a1373d3" containerName="authorino" Jul 09 18:36:16.089741 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.089683 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a8beed4-1bc6-46b5-afb5-02291f343f0d" containerName="authorino" Jul 09 18:36:16.089741 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.089688 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a8beed4-1bc6-46b5-afb5-02291f343f0d" containerName="authorino" Jul 09 18:36:16.089930 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.089757 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a8beed4-1bc6-46b5-afb5-02291f343f0d" containerName="authorino" Jul 09 18:36:16.089930 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.089766 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd7df1e-aa21-4067-9221-d3ad1a1373d3" containerName="authorino" Jul 09 18:36:16.089930 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.089774 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc9315d6-6ec9-40ee-b686-5ee41820e21e" containerName="authorino" Jul 09 18:36:16.092507 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.092491 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.096644 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.096614 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 18:36:16.096800 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.096662 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m2k7l\"" Jul 09 18:36:16.096800 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.096684 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 09 18:36:16.096800 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.096693 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 18:36:16.101950 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.101923 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs"] Jul 09 18:36:16.110387 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.110360 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:36:16.161305 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.161280 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22132870-ffb8-4340-a82c-3e327a8c7f3e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.161468 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.161310 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22132870-ffb8-4340-a82c-3e327a8c7f3e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.161468 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.161336 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22132870-ffb8-4340-a82c-3e327a8c7f3e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.161574 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.161458 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n85rz\" (UniqueName: \"kubernetes.io/projected/22132870-ffb8-4340-a82c-3e327a8c7f3e-kube-api-access-n85rz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.161574 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.161509 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22132870-ffb8-4340-a82c-3e327a8c7f3e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.161574 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.161564 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22132870-ffb8-4340-a82c-3e327a8c7f3e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.262666 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.262636 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22132870-ffb8-4340-a82c-3e327a8c7f3e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.262666 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.262669 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22132870-ffb8-4340-a82c-3e327a8c7f3e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.262887 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.262715 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22132870-ffb8-4340-a82c-3e327a8c7f3e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.262887 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.262765 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n85rz\" (UniqueName: \"kubernetes.io/projected/22132870-ffb8-4340-a82c-3e327a8c7f3e-kube-api-access-n85rz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.262887 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.262789 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22132870-ffb8-4340-a82c-3e327a8c7f3e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.262887 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.262812 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22132870-ffb8-4340-a82c-3e327a8c7f3e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.263137 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.263108 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22132870-ffb8-4340-a82c-3e327a8c7f3e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.263253 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.263161 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22132870-ffb8-4340-a82c-3e327a8c7f3e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.263253 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.263195 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22132870-ffb8-4340-a82c-3e327a8c7f3e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.264925 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.264898 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22132870-ffb8-4340-a82c-3e327a8c7f3e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.265185 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.265167 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22132870-ffb8-4340-a82c-3e327a8c7f3e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.271443 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.271417 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n85rz\" (UniqueName: \"kubernetes.io/projected/22132870-ffb8-4340-a82c-3e327a8c7f3e-kube-api-access-n85rz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l7svs\" (UID: \"22132870-ffb8-4340-a82c-3e327a8c7f3e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.402811 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.402775 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:16.531650 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.531511 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs"] Jul 09 18:36:16.533752 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:36:16.533727 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22132870_ffb8_4340_a82c_3e327a8c7f3e.slice/crio-166d58de74bf2b9ab89117c5328381a47dd1fe0f92d86b11937f3417830e57f7 WatchSource:0}: Error finding container 166d58de74bf2b9ab89117c5328381a47dd1fe0f92d86b11937f3417830e57f7: Status 404 returned error can't find the container with id 166d58de74bf2b9ab89117c5328381a47dd1fe0f92d86b11937f3417830e57f7 Jul 09 18:36:16.535150 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:16.535133 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 18:36:17.210686 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:17.210649 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" event={"ID":"22132870-ffb8-4340-a82c-3e327a8c7f3e","Type":"ContainerStarted","Data":"166d58de74bf2b9ab89117c5328381a47dd1fe0f92d86b11937f3417830e57f7"} Jul 09 18:36:18.216278 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:18.216238 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" event={"ID":"22132870-ffb8-4340-a82c-3e327a8c7f3e","Type":"ContainerStarted","Data":"4a6adcba2f82d786a810396940cb90174dea7a026b534e7c47f8bcae96cc1dec"} Jul 09 18:36:18.236564 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:18.236519 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" podStartSLOduration=1.215151167 podStartE2EDuration="2.236503953s" totalImagesPullingTime="1.021352786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:36:18.235898834 +0000 UTC m=+645.300564509" watchObservedRunningTime="2026-07-09 18:36:18.236503953 +0000 UTC m=+645.301169658" Jul 09 18:36:18.696864 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:18.696828 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:36:20.279713 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.279678 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2"] Jul 09 18:36:20.283175 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.283159 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.285982 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.285961 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 09 18:36:20.292954 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.292932 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2"] Jul 09 18:36:20.305971 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.305941 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:36:20.396041 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.396006 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39f7cd41-d094-447e-b7bf-ebc21d8bce35-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.396203 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.396050 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8z8x\" (UniqueName: \"kubernetes.io/projected/39f7cd41-d094-447e-b7bf-ebc21d8bce35-kube-api-access-f8z8x\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.396203 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.396140 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39f7cd41-d094-447e-b7bf-ebc21d8bce35-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.396203 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.396177 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39f7cd41-d094-447e-b7bf-ebc21d8bce35-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.396306 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.396222 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39f7cd41-d094-447e-b7bf-ebc21d8bce35-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.396306 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.396246 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39f7cd41-d094-447e-b7bf-ebc21d8bce35-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.497254 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.497222 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39f7cd41-d094-447e-b7bf-ebc21d8bce35-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.497365 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.497262 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39f7cd41-d094-447e-b7bf-ebc21d8bce35-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.497365 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.497305 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39f7cd41-d094-447e-b7bf-ebc21d8bce35-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.497365 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.497355 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39f7cd41-d094-447e-b7bf-ebc21d8bce35-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.497484 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.497390 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39f7cd41-d094-447e-b7bf-ebc21d8bce35-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.497484 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.497427 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8z8x\" (UniqueName: \"kubernetes.io/projected/39f7cd41-d094-447e-b7bf-ebc21d8bce35-kube-api-access-f8z8x\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.497720 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.497671 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39f7cd41-d094-447e-b7bf-ebc21d8bce35-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.497847 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.497743 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39f7cd41-d094-447e-b7bf-ebc21d8bce35-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.497847 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.497782 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39f7cd41-d094-447e-b7bf-ebc21d8bce35-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.499604 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.499582 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39f7cd41-d094-447e-b7bf-ebc21d8bce35-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.499805 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.499787 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39f7cd41-d094-447e-b7bf-ebc21d8bce35-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.506872 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.506852 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8z8x\" (UniqueName: \"kubernetes.io/projected/39f7cd41-d094-447e-b7bf-ebc21d8bce35-kube-api-access-f8z8x\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2\" (UID: \"39f7cd41-d094-447e-b7bf-ebc21d8bce35\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.591687 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.591594 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:20.721781 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:20.721743 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2"] Jul 09 18:36:20.725265 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:36:20.725234 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39f7cd41_d094_447e_b7bf_ebc21d8bce35.slice/crio-8317c86ef67dcb11692c17f29b7f272bb861dceecbe6a62d7b069f8cb6483553 WatchSource:0}: Error finding container 8317c86ef67dcb11692c17f29b7f272bb861dceecbe6a62d7b069f8cb6483553: Status 404 returned error can't find the container with id 8317c86ef67dcb11692c17f29b7f272bb861dceecbe6a62d7b069f8cb6483553 Jul 09 18:36:21.231218 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:21.231124 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" event={"ID":"39f7cd41-d094-447e-b7bf-ebc21d8bce35","Type":"ContainerStarted","Data":"266d81da3885e8fa36bc37f6d0f0d7ca244a7147cd65fd7e9508d970558a8166"} Jul 09 18:36:21.231218 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:21.231171 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" event={"ID":"39f7cd41-d094-447e-b7bf-ebc21d8bce35","Type":"ContainerStarted","Data":"8317c86ef67dcb11692c17f29b7f272bb861dceecbe6a62d7b069f8cb6483553"} Jul 09 18:36:21.252451 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:21.252405 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" podStartSLOduration=1.02319583 podStartE2EDuration="1.252389522s" totalImagesPullingTime="229.193692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:36:21.250526045 +0000 UTC m=+648.315191733" watchObservedRunningTime="2026-07-09 18:36:21.252389522 +0000 UTC m=+648.317055195" Jul 09 18:36:26.403270 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:26.403234 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:26.403804 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:26.403320 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:26.415600 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:26.415575 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:27.188956 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:27.188919 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:36:27.267978 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:27.267951 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l7svs" Jul 09 18:36:30.592570 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:30.592535 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:30.592570 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:30.592582 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:30.604947 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:30.604921 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:36:31.283832 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:36:31.283801 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2" Jul 09 18:37:24.643609 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:24.643578 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f89d6df55-z5nvk"] Jul 09 18:37:24.646860 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:24.646843 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f89d6df55-z5nvk" Jul 09 18:37:24.654866 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:24.654837 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f89d6df55-z5nvk"] Jul 09 18:37:24.732817 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:24.732787 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7777790b-7a7b-478b-b975-c0f16f8e7b5e-tls-cert\") pod \"authorino-5f89d6df55-z5nvk\" (UID: \"7777790b-7a7b-478b-b975-c0f16f8e7b5e\") " pod="kuadrant-system/authorino-5f89d6df55-z5nvk" Jul 09 18:37:24.732989 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:24.732824 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbmfx\" (UniqueName: \"kubernetes.io/projected/7777790b-7a7b-478b-b975-c0f16f8e7b5e-kube-api-access-wbmfx\") pod \"authorino-5f89d6df55-z5nvk\" (UID: \"7777790b-7a7b-478b-b975-c0f16f8e7b5e\") " pod="kuadrant-system/authorino-5f89d6df55-z5nvk" Jul 09 18:37:24.834092 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:24.834036 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7777790b-7a7b-478b-b975-c0f16f8e7b5e-tls-cert\") pod \"authorino-5f89d6df55-z5nvk\" (UID: \"7777790b-7a7b-478b-b975-c0f16f8e7b5e\") " pod="kuadrant-system/authorino-5f89d6df55-z5nvk" Jul 09 18:37:24.834092 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:24.834097 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbmfx\" (UniqueName: \"kubernetes.io/projected/7777790b-7a7b-478b-b975-c0f16f8e7b5e-kube-api-access-wbmfx\") pod \"authorino-5f89d6df55-z5nvk\" (UID: \"7777790b-7a7b-478b-b975-c0f16f8e7b5e\") " pod="kuadrant-system/authorino-5f89d6df55-z5nvk" Jul 09 18:37:24.836598 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:24.836574 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7777790b-7a7b-478b-b975-c0f16f8e7b5e-tls-cert\") pod \"authorino-5f89d6df55-z5nvk\" (UID: \"7777790b-7a7b-478b-b975-c0f16f8e7b5e\") " pod="kuadrant-system/authorino-5f89d6df55-z5nvk" Jul 09 18:37:24.841516 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:24.841489 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbmfx\" (UniqueName: \"kubernetes.io/projected/7777790b-7a7b-478b-b975-c0f16f8e7b5e-kube-api-access-wbmfx\") pod \"authorino-5f89d6df55-z5nvk\" (UID: \"7777790b-7a7b-478b-b975-c0f16f8e7b5e\") " pod="kuadrant-system/authorino-5f89d6df55-z5nvk" Jul 09 18:37:24.956098 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:24.956017 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f89d6df55-z5nvk" Jul 09 18:37:24.960657 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:24.960630 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:37:25.066837 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:25.066796 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:37:25.098623 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:25.098594 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f89d6df55-z5nvk"] Jul 09 18:37:25.101745 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:37:25.101672 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7777790b_7a7b_478b_b975_c0f16f8e7b5e.slice/crio-0f0d58649d9f860928b18bfcc49b4f02b08d62c8bf0e46a7e6e56d80c84893fe WatchSource:0}: Error finding container 0f0d58649d9f860928b18bfcc49b4f02b08d62c8bf0e46a7e6e56d80c84893fe: Status 404 returned error can't find the container with id 0f0d58649d9f860928b18bfcc49b4f02b08d62c8bf0e46a7e6e56d80c84893fe Jul 09 18:37:25.498611 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:25.498577 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f89d6df55-z5nvk" event={"ID":"7777790b-7a7b-478b-b975-c0f16f8e7b5e","Type":"ContainerStarted","Data":"0f0d58649d9f860928b18bfcc49b4f02b08d62c8bf0e46a7e6e56d80c84893fe"} Jul 09 18:37:26.194585 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:26.194539 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:37:26.336095 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:26.336060 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:37:26.506017 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:26.505922 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f89d6df55-z5nvk" event={"ID":"7777790b-7a7b-478b-b975-c0f16f8e7b5e","Type":"ContainerStarted","Data":"1e14f87fa6434d6137db225204cd27fc8e2ca41919bd8783886e3d862006ad55"} Jul 09 18:37:26.557775 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:26.557691 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f89d6df55-z5nvk" podStartSLOduration=1.8698078919999999 podStartE2EDuration="2.557670324s" totalImagesPullingTime="687.862432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:37:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:37:26.557130271 +0000 UTC m=+713.621795946" watchObservedRunningTime="2026-07-09 18:37:26.557670324 +0000 UTC m=+713.622335999" Jul 09 18:37:26.619601 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:26.619567 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77c849fddb-gkdt9"] Jul 09 18:37:26.619819 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:26.619786 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-77c849fddb-gkdt9" podUID="59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0" containerName="authorino" containerID="cri-o://3db30557017e9b5779dff71e84bc9826fee503874244350eafc3bbbc748a77b9" gracePeriod=30 Jul 09 18:37:26.887493 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:26.887463 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77c849fddb-gkdt9" Jul 09 18:37:26.954224 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:26.954193 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0-tls-cert\") pod \"59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0\" (UID: \"59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0\") " Jul 09 18:37:26.954224 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:26.954235 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nb4x\" (UniqueName: \"kubernetes.io/projected/59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0-kube-api-access-9nb4x\") pod \"59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0\" (UID: \"59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0\") " Jul 09 18:37:26.956319 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:26.956286 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0-kube-api-access-9nb4x" (OuterVolumeSpecName: "kube-api-access-9nb4x") pod "59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0" (UID: "59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0"). InnerVolumeSpecName "kube-api-access-9nb4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:37:26.964543 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:26.964515 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0" (UID: "59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:37:27.055563 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:27.055476 2558 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0-tls-cert\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:37:27.055563 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:27.055510 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nb4x\" (UniqueName: \"kubernetes.io/projected/59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0-kube-api-access-9nb4x\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:37:27.511757 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:27.511720 2558 generic.go:356] "Generic (PLEG): container finished" podID="59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0" containerID="3db30557017e9b5779dff71e84bc9826fee503874244350eafc3bbbc748a77b9" exitCode=0 Jul 09 18:37:27.512236 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:27.511770 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77c849fddb-gkdt9" Jul 09 18:37:27.512236 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:27.511795 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77c849fddb-gkdt9" event={"ID":"59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0","Type":"ContainerDied","Data":"3db30557017e9b5779dff71e84bc9826fee503874244350eafc3bbbc748a77b9"} Jul 09 18:37:27.512236 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:27.511840 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77c849fddb-gkdt9" event={"ID":"59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0","Type":"ContainerDied","Data":"e5fdfe30aa02264803416712cab13f6149f923acdb8eb78ce7caa8d1c77337f0"} Jul 09 18:37:27.512236 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:27.511859 2558 scope.go:117] "RemoveContainer" containerID="3db30557017e9b5779dff71e84bc9826fee503874244350eafc3bbbc748a77b9" Jul 09 18:37:27.521026 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:27.521008 2558 scope.go:117] "RemoveContainer" containerID="3db30557017e9b5779dff71e84bc9826fee503874244350eafc3bbbc748a77b9" Jul 09 18:37:27.521287 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:37:27.521267 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db30557017e9b5779dff71e84bc9826fee503874244350eafc3bbbc748a77b9\": container with ID starting with 3db30557017e9b5779dff71e84bc9826fee503874244350eafc3bbbc748a77b9 not found: ID does not exist" containerID="3db30557017e9b5779dff71e84bc9826fee503874244350eafc3bbbc748a77b9" Jul 09 18:37:27.521380 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:27.521294 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db30557017e9b5779dff71e84bc9826fee503874244350eafc3bbbc748a77b9"} err="failed to get container status \"3db30557017e9b5779dff71e84bc9826fee503874244350eafc3bbbc748a77b9\": rpc error: code = NotFound desc = could not find container \"3db30557017e9b5779dff71e84bc9826fee503874244350eafc3bbbc748a77b9\": container with ID starting with 3db30557017e9b5779dff71e84bc9826fee503874244350eafc3bbbc748a77b9 not found: ID does not exist" Jul 09 18:37:27.535328 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:27.535305 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77c849fddb-gkdt9"] Jul 09 18:37:27.539119 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:27.539098 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-77c849fddb-gkdt9"] Jul 09 18:37:27.559209 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:37:27.559187 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0" path="/var/lib/kubelet/pods/59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0/volumes" Jul 09 18:38:00.593402 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:38:00.593363 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:38:03.992148 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:38:03.992117 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:38:10.294172 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:38:10.294130 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:38:21.392861 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:38:21.392825 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:38:32.012783 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:38:32.012749 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:38:41.890974 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:38:41.890937 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:38:44.293297 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:38:44.293254 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:38:45.688048 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:38:45.688005 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:38:46.003508 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:38:46.003402 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:38:46.066613 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:38:46.066581 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:39:11.993958 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:39:11.993923 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:39:25.582487 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:39:25.582452 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:39:30.363335 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:39:30.363303 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:39:30.463354 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:39:30.463317 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:39:30.622157 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:39:30.622073 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:39:36.684547 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:39:36.684497 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:39:36.747692 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:39:36.747654 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:39:36.827710 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:39:36.827671 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:39:36.921211 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:39:36.921167 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:39:37.033030 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:39:37.032934 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:39:37.400080 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:39:37.400048 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:39:43.037833 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:39:43.037786 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:39:43.138500 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:39:43.138472 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:39:58.901382 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:39:58.901348 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:40:06.710121 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:40:06.710086 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:40:18.085909 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:40:18.085869 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:40:25.487339 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:40:25.487259 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:40:32.014824 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:40:32.014780 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:40:33.345195 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:40:33.345170 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:40:33.345622 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:40:33.345393 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:40:36.786577 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:40:36.786540 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:40:46.420059 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:40:46.420028 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:40:57.604633 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:40:57.604585 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:41:04.687460 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:41:04.687422 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:41:12.595969 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:41:12.595937 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:41:20.697613 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:41:20.697580 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:41:33.086553 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:41:33.086511 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:41:42.584280 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:41:42.584243 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:42:06.395327 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:42:06.395247 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:42:16.789061 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:42:16.789025 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:42:33.694952 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:42:33.694919 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:42:43.187114 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:42:43.187078 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:42:45.346835 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:42:45.346801 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:42:46.506799 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:42:46.506763 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:42:53.552463 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:42:53.552434 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:43:03.497650 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:43:03.497618 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:43:13.197665 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:43:13.197629 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:45:00.795727 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:45:00.795622 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:45:12.893648 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:45:12.893613 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:45:21.742152 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:45:21.742120 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:45:33.369290 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:45:33.369259 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:45:33.371724 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:45:33.371684 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:45:33.801056 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:45:33.801023 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:45:45.082994 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:45:45.082963 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:45:50.188672 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:45:50.188627 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:45:56.192683 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:45:56.192643 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:46:04.594392 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:46:04.594348 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:46:15.778863 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:46:15.778827 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:46:26.688367 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:46:26.688284 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:46:35.790813 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:46:35.790776 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:46:46.588595 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:46:46.588565 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:47:07.695882 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:47:07.695849 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:47:08.864244 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:47:08.864202 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:47:17.112794 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:47:17.112754 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:47:26.589472 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:47:26.589436 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:47:38.086395 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:47:38.086363 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:47:52.836387 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:47:52.836343 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:47:54.187447 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:47:54.187407 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:48:04.279482 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:48:04.279402 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:48:14.194507 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:48:14.194469 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:48:22.116656 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:48:22.116619 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:48:33.098733 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:48:33.098683 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:48:42.595569 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:48:42.595529 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:48:56.340953 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:48:56.340919 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:49:11.108007 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:49:11.107972 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:49:21.298948 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:49:21.298912 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:49:32.002569 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:49:32.002482 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:49:44.886569 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:49:44.886528 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:50:00.595714 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:00.595668 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:50:06.791264 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:06.791222 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:06.853241 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:06.853177 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:13.697311 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:13.697277 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:50:23.295915 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:23.295871 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:50:25.664560 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:25.664511 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:33.395013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:33.394986 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:50:33.397831 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:33.397809 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:50:33.850989 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:33.850903 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:35.887971 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:35.887937 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:50:41.467647 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:41.467613 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:46.202073 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:46.202040 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:50:46.624656 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:46.624577 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:49.717955 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:49.717914 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:49.822245 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:49.822210 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:50.202410 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:50.202378 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:53.921291 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:53.921257 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:50:56.499095 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:56.499002 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:56.600262 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:56.600229 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:56.726215 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:56.726181 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:56.783114 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:56.783016 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:57.898257 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:57.898220 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:57.971522 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:57.971486 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:58.035959 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:58.035924 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:58.267162 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:58.267087 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:58.482505 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:58.482472 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:50:58.585919 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:50:58.585815 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:00.733606 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:00.733574 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:02.900266 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:02.900213 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:03.098284 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:03.098246 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:03.193474 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:03.193362 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:03.299232 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:03.299197 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:04.184139 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:04.184098 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:51:06.504286 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:06.504253 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:09.451916 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:09.451874 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:09.515819 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:09.515787 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:09.707909 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:09.707821 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:09.806758 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:09.806694 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:09.945676 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:09.945647 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:10.041168 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:10.041081 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:11.237657 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:11.237627 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:11.649811 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:11.649778 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:11.775191 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:11.775152 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:12.435566 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:12.435535 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:12.534373 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:12.534338 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:13.261037 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:13.261001 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:13.681867 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:13.681832 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:51:15.666686 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:15.666644 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:15.867352 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:15.867316 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:15.967921 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:15.967827 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:16.258101 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:16.258020 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:19.235215 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:19.235185 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:22.706068 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:22.706030 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:22.777101 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:22.777071 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:22.943452 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:22.943413 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:23.000217 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:23.000144 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:24.116731 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:24.116671 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:24.198208 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:24.198174 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:24.265753 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:24.265694 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:24.533534 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:24.533494 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:24.853030 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:24.852950 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:24.952883 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:24.952847 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:26.032401 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:26.032345 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:27.493764 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:27.493733 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:51:29.205404 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:29.205370 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:29.648646 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:29.648608 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:35.335681 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:35.335634 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:35.436342 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:35.436305 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:39.588368 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:39.588334 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:51:42.858761 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:42.858720 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:42.930114 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:42.930075 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:43.171903 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:43.171865 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:43.275098 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:43.275062 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:44.321073 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:44.321037 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:44.433470 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:44.433435 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:44.608646 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:44.608555 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:44.982146 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:44.982110 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:45.395959 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:45.395929 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:45.497975 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:45.497942 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:46.363058 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:46.363028 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:48.093294 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:48.093255 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:51:48.711840 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:48.711803 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:48.901537 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:48.901499 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:49.002757 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:49.002663 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:49.173880 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:49.173838 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:50.364493 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:50.364458 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:56.840926 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:56.840889 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:56.915849 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:56.915815 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:57.384012 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:57.383966 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:57.470797 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:57.470760 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:51:58.980939 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:51:58.980898 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:52:01.229964 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:52:01.229930 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:52:06.145197 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:52:06.145154 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:52:06.222381 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:52:06.222341 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:52:06.352578 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:52:06.352544 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:52:06.441564 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:52:06.441472 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:52:06.888642 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:52:06.888612 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:52:32.875184 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:52:32.875089 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:52:39.139415 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:52:39.139376 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:52:39.211254 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:52:39.211221 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:52:39.315817 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:52:39.315784 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:52:55.236456 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:52:55.236424 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:01.731875 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:01.731839 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:01.803969 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:01.803934 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:01.917296 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:01.917255 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:02.013856 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:02.013760 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:12.503489 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:12.503454 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:13.464835 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:13.464794 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:19.988232 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:19.988187 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:20.109006 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:20.108967 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:20.220478 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:20.220434 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:20.324617 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:20.324525 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:21.187815 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:21.187778 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:21.444591 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:21.444506 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:21.548926 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:21.548898 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:22.384425 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:22.384393 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:53:23.128842 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:23.128808 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:29.205853 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:29.205821 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:29.271076 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:29.271040 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:29.884522 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:29.884485 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:30.012299 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:30.012268 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:53:36.000319 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:36.000282 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:36.100457 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:36.100419 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:36.238884 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:36.238845 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:36.598454 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:36.598418 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:36.926368 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:36.926330 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:39.275585 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:39.275541 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:39.298100 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:39.298060 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:53:39.361948 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:39.361914 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:39.447332 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:39.447305 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:39.854936 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:39.854908 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:39.957013 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:39.956977 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:43.838879 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:43.838842 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:44.547134 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:44.547098 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:44.663023 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:44.662986 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:44.947485 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:44.947444 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:49.993333 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:49.993298 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:53:51.491917 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:51.491878 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:51.563757 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:51.563723 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:51.940439 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:51.940403 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:52.341992 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:52.341899 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:52.498087 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:52.498044 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:57.468730 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:57.468632 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:53:58.947607 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:58.947570 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:59.068030 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:59.067994 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:59.214968 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:59.214878 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:59.435216 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:59.435170 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:59.570000 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:59.569907 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:59.656537 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:59.656500 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:53:59.774870 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:53:59.774837 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:05.732981 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:05.732947 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:05.839406 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:05.839367 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:05.933862 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:05.933823 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:06.081079 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:06.080996 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:10.543327 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:10.543292 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:10.655149 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:10.655098 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:12.912215 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:12.912181 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:13.010537 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:13.010504 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:19.836456 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:19.836416 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:20.133907 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:20.133866 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:20.333515 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:20.333475 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:23.453803 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:23.453764 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:23.556458 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:23.556427 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:23.633169 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:23.633132 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:23.748693 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:23.748608 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:24.179873 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:24.179843 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:24.279124 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:24.279089 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:27.943475 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:27.943425 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:29.291791 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:29.291757 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:54:55.694598 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.694567 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk"] Jul 09 18:54:55.695076 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.694880 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0" containerName="authorino" Jul 09 18:54:55.695076 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.694892 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0" containerName="authorino" Jul 09 18:54:55.695076 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.694948 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="59473d5a-f14e-4d6f-aa62-2bfbb01fb3a0" containerName="authorino" Jul 09 18:54:55.697864 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.697848 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.700577 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.700551 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8a4ed68f\"/\"kube-root-ca.crt\"" Jul 09 18:54:55.700577 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.700561 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8a4ed68f\"/\"openshift-service-ca.crt\"" Jul 09 18:54:55.701845 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.701810 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8a4ed68f\"/\"e2e-lifecycle-model-8a4ed68f-kserve-self-signed-certs\"" Jul 09 18:54:55.701845 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.701839 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8a4ed68f\"/\"default-dockercfg-48hrh\"" Jul 09 18:54:55.714927 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.714905 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk"] Jul 09 18:54:55.724531 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.724506 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:54:55.845431 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.845395 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-tmp-dir\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.845610 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.845439 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2mk5\" (UniqueName: \"kubernetes.io/projected/909db1d0-7fd7-4edd-9428-e3e81b4bae85-kube-api-access-m2mk5\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.845610 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.845473 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909db1d0-7fd7-4edd-9428-e3e81b4bae85-tls-certs\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.845610 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.845560 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-dshm\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.845610 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.845589 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-model-cache\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.845793 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.845620 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-home\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.946293 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.946208 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-dshm\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.946293 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.946251 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-model-cache\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.946293 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.946278 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-home\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.946537 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.946298 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-tmp-dir\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.946537 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.946332 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2mk5\" (UniqueName: \"kubernetes.io/projected/909db1d0-7fd7-4edd-9428-e3e81b4bae85-kube-api-access-m2mk5\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.946537 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.946360 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909db1d0-7fd7-4edd-9428-e3e81b4bae85-tls-certs\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.946730 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.946670 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-model-cache\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.946800 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.946759 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-home\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.946949 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.946926 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-tmp-dir\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.948737 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.948691 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-dshm\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.948952 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.948935 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909db1d0-7fd7-4edd-9428-e3e81b4bae85-tls-certs\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:55.955435 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:55.955412 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2mk5\" (UniqueName: \"kubernetes.io/projected/909db1d0-7fd7-4edd-9428-e3e81b4bae85-kube-api-access-m2mk5\") pod \"e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:56.007040 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:56.007010 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:54:56.143827 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:56.143792 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk"] Jul 09 18:54:56.147532 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:54:56.147503 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod909db1d0_7fd7_4edd_9428_e3e81b4bae85.slice/crio-c87459ad866934432cca97afa4202e81b2cc2d9ef8d080a4578cd068ed3afebc WatchSource:0}: Error finding container c87459ad866934432cca97afa4202e81b2cc2d9ef8d080a4578cd068ed3afebc: Status 404 returned error can't find the container with id c87459ad866934432cca97afa4202e81b2cc2d9ef8d080a4578cd068ed3afebc Jul 09 18:54:56.984726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:56.984663 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" event={"ID":"909db1d0-7fd7-4edd-9428-e3e81b4bae85","Type":"ContainerStarted","Data":"119ae44eb86ab8bea94d165eebadccd1013e3c0e3494f7db63259c4e996ec1e7"} Jul 09 18:54:56.984726 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:56.984728 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" event={"ID":"909db1d0-7fd7-4edd-9428-e3e81b4bae85","Type":"ContainerStarted","Data":"c87459ad866934432cca97afa4202e81b2cc2d9ef8d080a4578cd068ed3afebc"} Jul 09 18:54:57.002318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:57.002268 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" podStartSLOduration=2.002241679 podStartE2EDuration="2.002241679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:57.001770755 +0000 UTC m=+1764.066436432" watchObservedRunningTime="2026-07-09 18:54:57.002241679 +0000 UTC m=+1764.066907355" Jul 09 18:54:58.893764 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:54:58.893726 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:55:06.007808 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:06.007769 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:55:06.008251 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:06.007856 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:55:06.020128 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:06.020104 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:55:06.033893 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:06.033871 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:55:18.255016 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:18.254979 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:55:18.542285 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:18.542197 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:55:21.057055 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:21.057016 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:55:21.771003 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:21.770956 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:55:27.089457 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:27.089360 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:55:32.084323 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:32.084289 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk"] Jul 09 18:55:32.188631 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:32.188589 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:55:32.633958 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:32.633913 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:55:33.426474 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:33.426447 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:55:33.428080 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:33.428054 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 18:55:47.104378 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.104330 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" podUID="909db1d0-7fd7-4edd-9428-e3e81b4bae85" containerName="main" containerID="cri-o://119ae44eb86ab8bea94d165eebadccd1013e3c0e3494f7db63259c4e996ec1e7" gracePeriod=45 Jul 09 18:55:47.355640 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.355582 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:55:47.378038 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.378014 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2mk5\" (UniqueName: \"kubernetes.io/projected/909db1d0-7fd7-4edd-9428-e3e81b4bae85-kube-api-access-m2mk5\") pod \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " Jul 09 18:55:47.378120 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.378060 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-tmp-dir\") pod \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " Jul 09 18:55:47.378120 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.378079 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909db1d0-7fd7-4edd-9428-e3e81b4bae85-tls-certs\") pod \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " Jul 09 18:55:47.378215 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.378138 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-home\") pod \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " Jul 09 18:55:47.378215 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.378171 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-dshm\") pod \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " Jul 09 18:55:47.378292 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.378223 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-model-cache\") pod \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\" (UID: \"909db1d0-7fd7-4edd-9428-e3e81b4bae85\") " Jul 09 18:55:47.378344 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.378288 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "909db1d0-7fd7-4edd-9428-e3e81b4bae85" (UID: "909db1d0-7fd7-4edd-9428-e3e81b4bae85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:55:47.378403 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.378351 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-home" (OuterVolumeSpecName: "home") pod "909db1d0-7fd7-4edd-9428-e3e81b4bae85" (UID: "909db1d0-7fd7-4edd-9428-e3e81b4bae85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:55:47.378532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.378513 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-tmp-dir\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:55:47.378532 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.378532 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-home\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:55:47.378725 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.378538 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-model-cache" (OuterVolumeSpecName: "model-cache") pod "909db1d0-7fd7-4edd-9428-e3e81b4bae85" (UID: "909db1d0-7fd7-4edd-9428-e3e81b4bae85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:55:47.380227 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.380204 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/909db1d0-7fd7-4edd-9428-e3e81b4bae85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "909db1d0-7fd7-4edd-9428-e3e81b4bae85" (UID: "909db1d0-7fd7-4edd-9428-e3e81b4bae85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:55:47.380303 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.380216 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-dshm" (OuterVolumeSpecName: "dshm") pod "909db1d0-7fd7-4edd-9428-e3e81b4bae85" (UID: "909db1d0-7fd7-4edd-9428-e3e81b4bae85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:55:47.380303 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.380206 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909db1d0-7fd7-4edd-9428-e3e81b4bae85-kube-api-access-m2mk5" (OuterVolumeSpecName: "kube-api-access-m2mk5") pod "909db1d0-7fd7-4edd-9428-e3e81b4bae85" (UID: "909db1d0-7fd7-4edd-9428-e3e81b4bae85"). InnerVolumeSpecName "kube-api-access-m2mk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:55:47.479781 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.479746 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2mk5\" (UniqueName: \"kubernetes.io/projected/909db1d0-7fd7-4edd-9428-e3e81b4bae85-kube-api-access-m2mk5\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:55:47.479781 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.479776 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909db1d0-7fd7-4edd-9428-e3e81b4bae85-tls-certs\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:55:47.479781 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.479787 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-dshm\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:55:47.480137 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:47.479795 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909db1d0-7fd7-4edd-9428-e3e81b4bae85-model-cache\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:55:48.214451 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:48.214417 2558 generic.go:356] "Generic (PLEG): container finished" podID="909db1d0-7fd7-4edd-9428-e3e81b4bae85" containerID="119ae44eb86ab8bea94d165eebadccd1013e3c0e3494f7db63259c4e996ec1e7" exitCode=0 Jul 09 18:55:48.214892 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:48.214456 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" Jul 09 18:55:48.214892 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:48.214472 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" event={"ID":"909db1d0-7fd7-4edd-9428-e3e81b4bae85","Type":"ContainerDied","Data":"119ae44eb86ab8bea94d165eebadccd1013e3c0e3494f7db63259c4e996ec1e7"} Jul 09 18:55:48.214892 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:48.214507 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk" event={"ID":"909db1d0-7fd7-4edd-9428-e3e81b4bae85","Type":"ContainerDied","Data":"c87459ad866934432cca97afa4202e81b2cc2d9ef8d080a4578cd068ed3afebc"} Jul 09 18:55:48.214892 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:48.214526 2558 scope.go:117] "RemoveContainer" containerID="119ae44eb86ab8bea94d165eebadccd1013e3c0e3494f7db63259c4e996ec1e7" Jul 09 18:55:48.222657 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:48.222637 2558 scope.go:117] "RemoveContainer" containerID="119ae44eb86ab8bea94d165eebadccd1013e3c0e3494f7db63259c4e996ec1e7" Jul 09 18:55:48.222912 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:55:48.222893 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119ae44eb86ab8bea94d165eebadccd1013e3c0e3494f7db63259c4e996ec1e7\": container with ID starting with 119ae44eb86ab8bea94d165eebadccd1013e3c0e3494f7db63259c4e996ec1e7 not found: ID does not exist" containerID="119ae44eb86ab8bea94d165eebadccd1013e3c0e3494f7db63259c4e996ec1e7" Jul 09 18:55:48.222968 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:48.222921 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119ae44eb86ab8bea94d165eebadccd1013e3c0e3494f7db63259c4e996ec1e7"} err="failed to get container status \"119ae44eb86ab8bea94d165eebadccd1013e3c0e3494f7db63259c4e996ec1e7\": rpc error: code = NotFound desc = could not find container \"119ae44eb86ab8bea94d165eebadccd1013e3c0e3494f7db63259c4e996ec1e7\": container with ID starting with 119ae44eb86ab8bea94d165eebadccd1013e3c0e3494f7db63259c4e996ec1e7 not found: ID does not exist" Jul 09 18:55:48.236045 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:48.236022 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk"] Jul 09 18:55:48.240463 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:48.240441 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8a4ed68f/e2e-lifecycle-model-8a4ed68f-kserve-685699dff5-j22tk"] Jul 09 18:55:49.559112 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:55:49.559076 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="909db1d0-7fd7-4edd-9428-e3e81b4bae85" path="/var/lib/kubelet/pods/909db1d0-7fd7-4edd-9428-e3e81b4bae85/volumes" Jul 09 18:56:00.246049 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:00.246007 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:08.259167 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:08.259132 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:10.018074 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:10.018043 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:11.803254 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:11.803218 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:56:16.390632 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:16.390591 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:16.542274 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:16.542240 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:16.719688 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:16.719591 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:16.992122 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:16.992035 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:18.120525 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:18.120489 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:18.220552 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:18.220505 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:18.316015 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:18.315983 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:19.041556 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:19.041501 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:19.139111 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:19.139076 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:19.235576 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:19.235547 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:20.617119 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:20.617077 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:27.619172 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:27.619133 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:56:28.606610 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:28.606539 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:28.751709 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:28.751674 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:29.375021 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:29.374985 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:35.757374 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:35.757329 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:35.865976 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:35.865929 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:35.957038 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:35.956997 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:38.392154 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:38.392123 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:56:46.585318 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:46.585278 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:46.935927 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:46.935888 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:49.787076 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:49.787035 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:56:49.894537 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:56:49.894501 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:07.928673 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:07.928593 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:14.412521 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:14.412481 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:14.475771 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:14.475743 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:14.765152 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:14.765059 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:15.082524 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:15.082412 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:15.507938 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:15.507902 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:17.662752 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:17.662693 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:17.774953 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:17.774919 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:17.950221 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:17.950126 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:18.303539 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:18.303449 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:18.692979 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:18.692945 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:18.792054 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:18.792021 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:22.176713 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:22.176658 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:22.506340 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:22.506247 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:22.938000 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:22.937966 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:23.317924 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:23.317842 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:25.701441 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:25.701401 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:25.802437 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:25.802401 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:25.835627 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:25.835591 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:57:25.872755 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:25.872725 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:25.972097 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:25.972008 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:26.701866 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:26.701835 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:26.801473 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:26.801437 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:30.205014 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:30.204980 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:40.876343 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:40.876292 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:40.970733 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:40.970668 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:41.107639 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:41.107606 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:57:47.827055 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:47.827016 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:47.909112 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:47.909053 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:51.414468 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.414423 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b"] Jul 09 18:57:51.414967 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.414896 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="909db1d0-7fd7-4edd-9428-e3e81b4bae85" containerName="main" Jul 09 18:57:51.414967 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.414917 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="909db1d0-7fd7-4edd-9428-e3e81b4bae85" containerName="main" Jul 09 18:57:51.415077 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.415038 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="909db1d0-7fd7-4edd-9428-e3e81b4bae85" containerName="main" Jul 09 18:57:51.418057 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.418030 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.422693 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.422670 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-15a9af\"/\"default-dockercfg-ts4dn\"" Jul 09 18:57:51.422854 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.422674 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-15a9af\"/\"openshift-service-ca.crt\"" Jul 09 18:57:51.423868 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.423845 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-15a9af\"/\"test-model-15a9af-kserve-self-signed-certs\"" Jul 09 18:57:51.423990 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.423963 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-15a9af\"/\"kube-root-ca.crt\"" Jul 09 18:57:51.429643 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.429615 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b"] Jul 09 18:57:51.459033 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.458987 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:57:51.498192 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.498158 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-tmp-dir\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.498192 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.498196 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-dshm\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.498377 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.498213 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-tls-certs\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.498377 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.498304 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-model-cache\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.498377 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.498356 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crhmp\" (UniqueName: \"kubernetes.io/projected/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-kube-api-access-crhmp\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.498480 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.498380 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-home\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.599485 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.599448 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-tmp-dir\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.599485 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.599489 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-dshm\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.599695 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.599504 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-tls-certs\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.599695 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.599530 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-model-cache\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.599829 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.599738 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crhmp\" (UniqueName: \"kubernetes.io/projected/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-kube-api-access-crhmp\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.599829 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.599800 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-home\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.599938 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.599922 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-tmp-dir\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.599999 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.599968 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-model-cache\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.600135 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.600114 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-home\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.601734 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.601714 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-dshm\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.602060 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.602043 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-tls-certs\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.613248 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.613225 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crhmp\" (UniqueName: \"kubernetes.io/projected/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-kube-api-access-crhmp\") pod \"test-model-15a9af-kserve-78ff5cfbc5-8rv7b\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.727968 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.727878 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:57:51.880772 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:51.880737 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b"] Jul 09 18:57:51.884833 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:57:51.884806 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ef7e8d1_9e46_4a54_a2a4_ff9e847dbac7.slice/crio-1526e61c523bcc66d829294339efefcbb6dea4f04a7253d5c4c32e42e58c08a1 WatchSource:0}: Error finding container 1526e61c523bcc66d829294339efefcbb6dea4f04a7253d5c4c32e42e58c08a1: Status 404 returned error can't find the container with id 1526e61c523bcc66d829294339efefcbb6dea4f04a7253d5c4c32e42e58c08a1 Jul 09 18:57:52.741984 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:52.741946 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" event={"ID":"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7","Type":"ContainerStarted","Data":"bd8cc018abf4e1a23955aaed41e3fb1c27e311c5a51176cb5166f71df324f6e9"} Jul 09 18:57:52.741984 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:52.741988 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" event={"ID":"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7","Type":"ContainerStarted","Data":"1526e61c523bcc66d829294339efefcbb6dea4f04a7253d5c4c32e42e58c08a1"} Jul 09 18:57:52.767269 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:52.767227 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" podStartSLOduration=1.767211935 podStartE2EDuration="1.767211935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:57:52.765394699 +0000 UTC m=+1939.830060372" watchObservedRunningTime="2026-07-09 18:57:52.767211935 +0000 UTC m=+1939.831877626" Jul 09 18:57:55.010856 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:57:55.010827 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:58:01.728424 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:01.728384 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:58:01.728923 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:01.728444 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:58:01.741238 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:01.741209 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:58:01.799357 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:01.799330 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:58:08.993105 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:08.993070 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:58:13.990285 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:13.990247 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p"] Jul 09 18:58:13.993473 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:13.993450 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:13.996953 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:13.996934 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-86d225\"/\"default-dockercfg-xstb8\"" Jul 09 18:58:13.997083 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:13.997001 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-86d225\"/\"test-model-86d225-kserve-self-signed-certs\"" Jul 09 18:58:13.997240 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:13.997222 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-86d225\"/\"kube-root-ca.crt\"" Jul 09 18:58:13.997306 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:13.997253 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-86d225\"/\"openshift-service-ca.crt\"" Jul 09 18:58:14.004397 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.004375 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p"] Jul 09 18:58:14.023478 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.023452 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:58:14.089249 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.089225 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-model-cache\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.089343 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.089264 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-tmp-dir\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.089343 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.089301 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b558580-3135-4f9a-8208-d13e798a61d7-tls-certs\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.089427 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.089354 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mdkf\" (UniqueName: \"kubernetes.io/projected/1b558580-3135-4f9a-8208-d13e798a61d7-kube-api-access-8mdkf\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.089427 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.089386 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-dshm\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.089427 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.089412 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-home\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.190094 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.190058 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-home\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.190194 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.190107 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-model-cache\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.190194 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.190142 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-tmp-dir\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.190194 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.190184 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b558580-3135-4f9a-8208-d13e798a61d7-tls-certs\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.190306 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.190204 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mdkf\" (UniqueName: \"kubernetes.io/projected/1b558580-3135-4f9a-8208-d13e798a61d7-kube-api-access-8mdkf\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.190306 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.190223 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-dshm\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.190528 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.190506 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-home\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.190577 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.190543 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-tmp-dir\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.190617 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.190577 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-model-cache\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.192533 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.192515 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-dshm\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.192632 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.192616 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b558580-3135-4f9a-8208-d13e798a61d7-tls-certs\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.200382 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.200356 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mdkf\" (UniqueName: \"kubernetes.io/projected/1b558580-3135-4f9a-8208-d13e798a61d7-kube-api-access-8mdkf\") pod \"test-model-86d225-kserve-d54d976c-wcx7p\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.302633 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.302561 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:14.444331 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.444307 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p"] Jul 09 18:58:14.446642 ip-10-0-138-253 kubenswrapper[2558]: W0709 18:58:14.446596 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b558580_3135_4f9a_8208_d13e798a61d7.slice/crio-d6ccddc3db212b88a96d381863157c26efa1fab7ed8a3b3f97a44919edf2b59b WatchSource:0}: Error finding container d6ccddc3db212b88a96d381863157c26efa1fab7ed8a3b3f97a44919edf2b59b: Status 404 returned error can't find the container with id d6ccddc3db212b88a96d381863157c26efa1fab7ed8a3b3f97a44919edf2b59b Jul 09 18:58:14.844673 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.844639 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" event={"ID":"1b558580-3135-4f9a-8208-d13e798a61d7","Type":"ContainerStarted","Data":"39f6b76bca72e10439df61c2bd36533e14492628b690964f389c57ee3c9af919"} Jul 09 18:58:14.844872 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.844681 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" event={"ID":"1b558580-3135-4f9a-8208-d13e798a61d7","Type":"ContainerStarted","Data":"d6ccddc3db212b88a96d381863157c26efa1fab7ed8a3b3f97a44919edf2b59b"} Jul 09 18:58:14.868043 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:14.867996 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" podStartSLOduration=1.86798173 podStartE2EDuration="1.86798173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:14.866130717 +0000 UTC m=+1961.930796389" watchObservedRunningTime="2026-07-09 18:58:14.86798173 +0000 UTC m=+1961.932647405" Jul 09 18:58:19.621580 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:19.621540 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:58:24.303412 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:24.303368 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:24.303886 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:24.303431 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:24.316078 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:24.316052 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:24.899877 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:24.899842 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:35.406276 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:35.406244 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:58:39.189797 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:39.189762 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b"] Jul 09 18:58:39.205273 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:39.205237 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:58:40.078034 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:40.077997 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p"] Jul 09 18:58:40.100911 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:40.100881 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:58:40.271715 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:40.271671 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:58:40.691672 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:40.691642 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:58:40.803311 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:40.803276 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:58:40.922103 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:40.922070 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:58:41.180569 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:41.180537 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:58:45.745751 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:45.745715 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:58:52.820030 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:52.819989 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:58:54.209344 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.209292 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" podUID="5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7" containerName="main" containerID="cri-o://bd8cc018abf4e1a23955aaed41e3fb1c27e311c5a51176cb5166f71df324f6e9" gracePeriod=45 Jul 09 18:58:54.468083 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.468027 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:58:54.532681 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.532653 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-tmp-dir\") pod \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " Jul 09 18:58:54.532841 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.532731 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-model-cache\") pod \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " Jul 09 18:58:54.532841 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.532778 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-dshm\") pod \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " Jul 09 18:58:54.532841 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.532793 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-home\") pod \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " Jul 09 18:58:54.532841 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.532825 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crhmp\" (UniqueName: \"kubernetes.io/projected/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-kube-api-access-crhmp\") pod \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " Jul 09 18:58:54.533070 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.532855 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-tls-certs\") pod \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\" (UID: \"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7\") " Jul 09 18:58:54.533070 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.532944 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7" (UID: "5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:58:54.533070 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.533032 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7" (UID: "5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:58:54.533210 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.533076 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-home" (OuterVolumeSpecName: "home") pod "5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7" (UID: "5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:58:54.533210 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.533160 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-tmp-dir\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:58:54.533210 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.533176 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-model-cache\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:58:54.533210 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.533184 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-home\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:58:54.534993 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.534959 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-dshm" (OuterVolumeSpecName: "dshm") pod "5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7" (UID: "5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:58:54.535107 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.535083 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7" (UID: "5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:58:54.535107 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.535095 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-kube-api-access-crhmp" (OuterVolumeSpecName: "kube-api-access-crhmp") pod "5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7" (UID: "5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7"). InnerVolumeSpecName "kube-api-access-crhmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:58:54.633890 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.633853 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-dshm\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:58:54.633890 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.633882 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crhmp\" (UniqueName: \"kubernetes.io/projected/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-kube-api-access-crhmp\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:58:54.633890 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:54.633896 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7-tls-certs\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:58:55.024628 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.024594 2558 generic.go:356] "Generic (PLEG): container finished" podID="5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7" containerID="bd8cc018abf4e1a23955aaed41e3fb1c27e311c5a51176cb5166f71df324f6e9" exitCode=0 Jul 09 18:58:55.024827 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.024637 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" Jul 09 18:58:55.024827 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.024674 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" event={"ID":"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7","Type":"ContainerDied","Data":"bd8cc018abf4e1a23955aaed41e3fb1c27e311c5a51176cb5166f71df324f6e9"} Jul 09 18:58:55.024827 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.024744 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b" event={"ID":"5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7","Type":"ContainerDied","Data":"1526e61c523bcc66d829294339efefcbb6dea4f04a7253d5c4c32e42e58c08a1"} Jul 09 18:58:55.024827 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.024767 2558 scope.go:117] "RemoveContainer" containerID="bd8cc018abf4e1a23955aaed41e3fb1c27e311c5a51176cb5166f71df324f6e9" Jul 09 18:58:55.033778 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.033761 2558 scope.go:117] "RemoveContainer" containerID="bd8cc018abf4e1a23955aaed41e3fb1c27e311c5a51176cb5166f71df324f6e9" Jul 09 18:58:55.034121 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:58:55.034101 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8cc018abf4e1a23955aaed41e3fb1c27e311c5a51176cb5166f71df324f6e9\": container with ID starting with bd8cc018abf4e1a23955aaed41e3fb1c27e311c5a51176cb5166f71df324f6e9 not found: ID does not exist" containerID="bd8cc018abf4e1a23955aaed41e3fb1c27e311c5a51176cb5166f71df324f6e9" Jul 09 18:58:55.034206 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.034136 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8cc018abf4e1a23955aaed41e3fb1c27e311c5a51176cb5166f71df324f6e9"} err="failed to get container status \"bd8cc018abf4e1a23955aaed41e3fb1c27e311c5a51176cb5166f71df324f6e9\": rpc error: code = NotFound desc = could not find container \"bd8cc018abf4e1a23955aaed41e3fb1c27e311c5a51176cb5166f71df324f6e9\": container with ID starting with bd8cc018abf4e1a23955aaed41e3fb1c27e311c5a51176cb5166f71df324f6e9 not found: ID does not exist" Jul 09 18:58:55.051534 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.051487 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b"] Jul 09 18:58:55.053603 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.053569 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-15a9af/test-model-15a9af-kserve-78ff5cfbc5-8rv7b"] Jul 09 18:58:55.097819 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.097768 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" podUID="1b558580-3135-4f9a-8208-d13e798a61d7" containerName="main" containerID="cri-o://39f6b76bca72e10439df61c2bd36533e14492628b690964f389c57ee3c9af919" gracePeriod=45 Jul 09 18:58:55.343944 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.343919 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:55.443452 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.443415 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-tmp-dir\") pod \"1b558580-3135-4f9a-8208-d13e798a61d7\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " Jul 09 18:58:55.443452 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.443456 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-dshm\") pod \"1b558580-3135-4f9a-8208-d13e798a61d7\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " Jul 09 18:58:55.443722 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.443504 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-home\") pod \"1b558580-3135-4f9a-8208-d13e798a61d7\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " Jul 09 18:58:55.443722 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.443526 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b558580-3135-4f9a-8208-d13e798a61d7-tls-certs\") pod \"1b558580-3135-4f9a-8208-d13e798a61d7\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " Jul 09 18:58:55.443722 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.443585 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-model-cache\") pod \"1b558580-3135-4f9a-8208-d13e798a61d7\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " Jul 09 18:58:55.443722 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.443617 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mdkf\" (UniqueName: \"kubernetes.io/projected/1b558580-3135-4f9a-8208-d13e798a61d7-kube-api-access-8mdkf\") pod \"1b558580-3135-4f9a-8208-d13e798a61d7\" (UID: \"1b558580-3135-4f9a-8208-d13e798a61d7\") " Jul 09 18:58:55.443931 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.443769 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b558580-3135-4f9a-8208-d13e798a61d7" (UID: "1b558580-3135-4f9a-8208-d13e798a61d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:58:55.443931 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.443799 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-home" (OuterVolumeSpecName: "home") pod "1b558580-3135-4f9a-8208-d13e798a61d7" (UID: "1b558580-3135-4f9a-8208-d13e798a61d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:58:55.443931 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.443896 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b558580-3135-4f9a-8208-d13e798a61d7" (UID: "1b558580-3135-4f9a-8208-d13e798a61d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:58:55.443931 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.443905 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-home\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:58:55.444118 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.443945 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-tmp-dir\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:58:55.445856 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.445824 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b558580-3135-4f9a-8208-d13e798a61d7-kube-api-access-8mdkf" (OuterVolumeSpecName: "kube-api-access-8mdkf") pod "1b558580-3135-4f9a-8208-d13e798a61d7" (UID: "1b558580-3135-4f9a-8208-d13e798a61d7"). InnerVolumeSpecName "kube-api-access-8mdkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:58:55.445856 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.445839 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b558580-3135-4f9a-8208-d13e798a61d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b558580-3135-4f9a-8208-d13e798a61d7" (UID: "1b558580-3135-4f9a-8208-d13e798a61d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:58:55.446003 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.445869 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-dshm" (OuterVolumeSpecName: "dshm") pod "1b558580-3135-4f9a-8208-d13e798a61d7" (UID: "1b558580-3135-4f9a-8208-d13e798a61d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:58:55.545369 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.545329 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b558580-3135-4f9a-8208-d13e798a61d7-tls-certs\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:58:55.545369 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.545363 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-model-cache\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:58:55.545369 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.545375 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mdkf\" (UniqueName: \"kubernetes.io/projected/1b558580-3135-4f9a-8208-d13e798a61d7-kube-api-access-8mdkf\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:58:55.545587 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.545390 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b558580-3135-4f9a-8208-d13e798a61d7-dshm\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 18:58:55.558604 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:55.558567 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7" path="/var/lib/kubelet/pods/5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7/volumes" Jul 09 18:58:56.030964 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:56.030928 2558 generic.go:356] "Generic (PLEG): container finished" podID="1b558580-3135-4f9a-8208-d13e798a61d7" containerID="39f6b76bca72e10439df61c2bd36533e14492628b690964f389c57ee3c9af919" exitCode=0 Jul 09 18:58:56.031140 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:56.030978 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" Jul 09 18:58:56.031140 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:56.031013 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" event={"ID":"1b558580-3135-4f9a-8208-d13e798a61d7","Type":"ContainerDied","Data":"39f6b76bca72e10439df61c2bd36533e14492628b690964f389c57ee3c9af919"} Jul 09 18:58:56.031140 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:56.031058 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p" event={"ID":"1b558580-3135-4f9a-8208-d13e798a61d7","Type":"ContainerDied","Data":"d6ccddc3db212b88a96d381863157c26efa1fab7ed8a3b3f97a44919edf2b59b"} Jul 09 18:58:56.031140 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:56.031079 2558 scope.go:117] "RemoveContainer" containerID="39f6b76bca72e10439df61c2bd36533e14492628b690964f389c57ee3c9af919" Jul 09 18:58:56.039113 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:56.039094 2558 scope.go:117] "RemoveContainer" containerID="39f6b76bca72e10439df61c2bd36533e14492628b690964f389c57ee3c9af919" Jul 09 18:58:56.039370 ip-10-0-138-253 kubenswrapper[2558]: E0709 18:58:56.039353 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f6b76bca72e10439df61c2bd36533e14492628b690964f389c57ee3c9af919\": container with ID starting with 39f6b76bca72e10439df61c2bd36533e14492628b690964f389c57ee3c9af919 not found: ID does not exist" containerID="39f6b76bca72e10439df61c2bd36533e14492628b690964f389c57ee3c9af919" Jul 09 18:58:56.039416 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:56.039379 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f6b76bca72e10439df61c2bd36533e14492628b690964f389c57ee3c9af919"} err="failed to get container status \"39f6b76bca72e10439df61c2bd36533e14492628b690964f389c57ee3c9af919\": rpc error: code = NotFound desc = could not find container \"39f6b76bca72e10439df61c2bd36533e14492628b690964f389c57ee3c9af919\": container with ID starting with 39f6b76bca72e10439df61c2bd36533e14492628b690964f389c57ee3c9af919 not found: ID does not exist" Jul 09 18:58:56.055773 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:56.055744 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p"] Jul 09 18:58:56.060171 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:56.060148 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-86d225/test-model-86d225-kserve-d54d976c-wcx7p"] Jul 09 18:58:57.558767 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:58:57.558737 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b558580-3135-4f9a-8208-d13e798a61d7" path="/var/lib/kubelet/pods/1b558580-3135-4f9a-8208-d13e798a61d7/volumes" Jul 09 18:59:11.588415 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:11.588347 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:11.687079 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:11.687043 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:23.594179 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:23.594145 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:23.911852 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:23.911816 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:24.113023 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:24.112983 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:24.322739 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:24.322643 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:24.692352 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:24.692321 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:24.793810 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:24.793781 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:24.896742 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:24.896686 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:25.123369 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:25.123285 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:28.395643 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:28.395610 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:59:29.691086 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:29.691053 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:29.761762 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:29.761725 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:31.511216 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:31.511174 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:31.614985 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:31.614945 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:31.894596 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:31.894562 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:32.098644 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:32.098616 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:32.444588 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:32.444554 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:32.829141 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:32.829058 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:32.923721 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:32.923674 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:36.092906 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:36.092860 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:36.209134 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:36.209094 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:36.301043 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:36.300999 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:36.619455 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:36.619420 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:36.918068 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:36.918030 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:37.048977 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:37.048945 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:40.509017 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:40.508983 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:41.018017 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:41.017978 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:41.330829 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:41.330745 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:41.712868 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:41.712831 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:41.795652 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:41.795616 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:59:44.571050 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:44.571016 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:44.670610 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:44.670579 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:44.779917 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:44.779879 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:45.249326 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:45.249299 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:45.414881 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:45.414847 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:45.514510 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:45.514421 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:49.203151 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:49.203112 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:57.194592 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:57.194507 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:57.590202 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:57.590097 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:57.811913 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:57.811875 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:57.912540 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:57.912494 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 18:59:58.013422 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:58.013377 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 18:59:58.119817 ip-10-0-138-253 kubenswrapper[2558]: I0709 18:59:58.119777 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:04.186164 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:04.186125 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:04.285194 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:04.285154 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:05.212013 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:05.211973 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:05.313007 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:05.312963 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:05.419177 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:05.419137 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:05.818985 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:05.818949 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:06.137448 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:06.137419 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:10.292759 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:10.292725 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:00:12.150910 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:12.150877 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:12.247210 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:12.247175 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:12.408945 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:12.408856 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:12.626403 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:12.626370 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:12.964070 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:12.964040 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:13.308511 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:13.308414 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:13.583620 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:13.583540 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:16.166778 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:16.166743 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:16.258863 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:16.258831 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:16.364379 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:16.364340 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:16.607978 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:16.607884 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:17.278194 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:17.278152 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:17.381597 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:17.381562 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:20.664963 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:20.664923 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:21.036339 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:21.036256 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:21.458905 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:21.458868 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:21.754044 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:21.753957 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:22.132187 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:22.132163 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:00:24.830111 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:24.830078 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:24.923993 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:24.923957 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:25.214216 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:25.214176 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:25.714189 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:25.714158 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:25.925991 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:25.925956 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:26.025801 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:26.025716 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:30.038563 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:30.038521 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:31.612522 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:31.612490 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:00:33.458873 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:33.458835 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 19:00:33.460353 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:33.460330 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 19:00:37.992942 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:37.992906 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:38.092624 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:38.092594 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:46.507518 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.507485 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br"] Jul 09 19:00:46.508042 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.507817 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b558580-3135-4f9a-8208-d13e798a61d7" containerName="main" Jul 09 19:00:46.508042 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.507828 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b558580-3135-4f9a-8208-d13e798a61d7" containerName="main" Jul 09 19:00:46.508042 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.507845 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7" containerName="main" Jul 09 19:00:46.508042 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.507853 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7" containerName="main" Jul 09 19:00:46.508042 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.507934 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef7e8d1-9e46-4a54-a2a4-ff9e847dbac7" containerName="main" Jul 09 19:00:46.508042 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.507946 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b558580-3135-4f9a-8208-d13e798a61d7" containerName="main" Jul 09 19:00:46.511162 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.511144 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.514891 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.514871 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4805f3\"/\"openshift-service-ca.crt\"" Jul 09 19:00:46.515029 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.514900 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4805f3\"/\"kube-root-ca.crt\"" Jul 09 19:00:46.515029 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.514904 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4805f3\"/\"default-dockercfg-97whk\"" Jul 09 19:00:46.515029 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.514904 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4805f3\"/\"auth-test-model-4805f3-kserve-self-signed-certs\"" Jul 09 19:00:46.529562 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.529540 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br"] Jul 09 19:00:46.539514 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.539489 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:46.687949 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.687893 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-home\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.687949 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.687960 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a985ae-bea3-4852-ab8a-844ce1158228-tls-certs\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.688227 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.688004 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-tmp-dir\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.688227 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.688027 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-model-cache\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.688227 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.688056 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-dshm\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.688227 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.688144 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kxm6\" (UniqueName: \"kubernetes.io/projected/95a985ae-bea3-4852-ab8a-844ce1158228-kube-api-access-9kxm6\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.789102 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.789005 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-home\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.789102 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.789055 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a985ae-bea3-4852-ab8a-844ce1158228-tls-certs\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.789102 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.789082 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-tmp-dir\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.789374 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.789114 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-model-cache\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.789374 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.789140 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-dshm\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.789374 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.789192 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kxm6\" (UniqueName: \"kubernetes.io/projected/95a985ae-bea3-4852-ab8a-844ce1158228-kube-api-access-9kxm6\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.789516 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.789473 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-home\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.789516 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.789495 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-tmp-dir\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.789610 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.789547 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-model-cache\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.791492 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.791470 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-dshm\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.791657 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.791640 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a985ae-bea3-4852-ab8a-844ce1158228-tls-certs\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.798172 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.798150 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kxm6\" (UniqueName: \"kubernetes.io/projected/95a985ae-bea3-4852-ab8a-844ce1158228-kube-api-access-9kxm6\") pod \"auth-test-model-4805f3-kserve-796b65875f-9c5br\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.820077 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.820048 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:46.952864 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:46.952825 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br"] Jul 09 19:00:46.955998 ip-10-0-138-253 kubenswrapper[2558]: W0709 19:00:46.955968 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95a985ae_bea3_4852_ab8a_844ce1158228.slice/crio-fea6cf82c58e974fe08899f492030771b0a93ca3394389c16568bbb1672c7b6b WatchSource:0}: Error finding container fea6cf82c58e974fe08899f492030771b0a93ca3394389c16568bbb1672c7b6b: Status 404 returned error can't find the container with id fea6cf82c58e974fe08899f492030771b0a93ca3394389c16568bbb1672c7b6b Jul 09 19:00:47.514604 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:47.514571 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" event={"ID":"95a985ae-bea3-4852-ab8a-844ce1158228","Type":"ContainerStarted","Data":"e0b55783b5fb709b08ee56bbd144c945c7a7bcdb9d7566f3a82786dfb3433d46"} Jul 09 19:00:47.514993 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:47.514617 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" event={"ID":"95a985ae-bea3-4852-ab8a-844ce1158228","Type":"ContainerStarted","Data":"fea6cf82c58e974fe08899f492030771b0a93ca3394389c16568bbb1672c7b6b"} Jul 09 19:00:47.538206 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:47.538161 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" podStartSLOduration=1.538147341 podStartE2EDuration="1.538147341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:47.536851222 +0000 UTC m=+2114.601516890" watchObservedRunningTime="2026-07-09 19:00:47.538147341 +0000 UTC m=+2114.602813015" Jul 09 19:00:47.835711 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:47.835619 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:47.935405 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:47.935370 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:48.331923 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:48.331886 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:48.424463 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:48.424431 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:00:50.302252 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:50.302218 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:00:56.820496 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:56.820456 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:56.820944 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:56.820726 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:56.833141 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:56.833120 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:57.568315 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:57.568284 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:00:59.151681 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:00:59.151640 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:01:21.626094 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.626059 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7"] Jul 09 19:01:21.629429 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.629414 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.634598 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.634547 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-172f88\"/\"kube-root-ca.crt\"" Jul 09 19:01:21.634598 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.634547 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-172f88\"/\"openshift-service-ca.crt\"" Jul 09 19:01:21.634820 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.634604 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-172f88\"/\"auth-test-model-172f88-kserve-self-signed-certs\"" Jul 09 19:01:21.634820 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.634564 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-172f88\"/\"default-dockercfg-894hs\"" Jul 09 19:01:21.648380 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.648354 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7"] Jul 09 19:01:21.680277 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.680249 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3de87bd-6eba-4c63-a262-0baef48813cc-tls-certs\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.680423 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.680285 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-home\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.680423 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.680313 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-dshm\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.680423 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.680386 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-tmp-dir\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.680528 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.680513 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-model-cache\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.680570 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.680530 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdl8\" (UniqueName: \"kubernetes.io/projected/e3de87bd-6eba-4c63-a262-0baef48813cc-kube-api-access-4jdl8\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.703423 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.703382 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:01:21.781892 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.781852 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-tmp-dir\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.782081 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.781944 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-model-cache\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.782081 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.781968 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdl8\" (UniqueName: \"kubernetes.io/projected/e3de87bd-6eba-4c63-a262-0baef48813cc-kube-api-access-4jdl8\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.782081 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.782000 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3de87bd-6eba-4c63-a262-0baef48813cc-tls-certs\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.782081 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.782027 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-home\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.782310 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.782246 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-dshm\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.782310 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.782273 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-tmp-dir\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.782415 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.782321 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-model-cache\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.782415 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.782379 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-home\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.784361 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.784328 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-dshm\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.784585 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.784566 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3de87bd-6eba-4c63-a262-0baef48813cc-tls-certs\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.793353 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.793332 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdl8\" (UniqueName: \"kubernetes.io/projected/e3de87bd-6eba-4c63-a262-0baef48813cc-kube-api-access-4jdl8\") pod \"auth-test-model-172f88-kserve-69d6586999-mm4x7\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:21.939237 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:21.939196 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:22.072483 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:22.072453 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7"] Jul 09 19:01:22.075977 ip-10-0-138-253 kubenswrapper[2558]: W0709 19:01:22.075953 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3de87bd_6eba_4c63_a262_0baef48813cc.slice/crio-9b27f5abf1535edd64267a660d82cabf018c499909cf6a889e5bfbdd4a2d1f8e WatchSource:0}: Error finding container 9b27f5abf1535edd64267a660d82cabf018c499909cf6a889e5bfbdd4a2d1f8e: Status 404 returned error can't find the container with id 9b27f5abf1535edd64267a660d82cabf018c499909cf6a889e5bfbdd4a2d1f8e Jul 09 19:01:22.664472 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:22.664432 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" event={"ID":"e3de87bd-6eba-4c63-a262-0baef48813cc","Type":"ContainerStarted","Data":"afd728069dce6a7ff51a093f0240afd223e3aae66624437e0d544db54c9c8811"} Jul 09 19:01:22.664472 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:22.664480 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" event={"ID":"e3de87bd-6eba-4c63-a262-0baef48813cc","Type":"ContainerStarted","Data":"9b27f5abf1535edd64267a660d82cabf018c499909cf6a889e5bfbdd4a2d1f8e"} Jul 09 19:01:22.687953 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:22.687904 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" podStartSLOduration=1.687889238 podStartE2EDuration="1.687889238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:01:22.687220382 +0000 UTC m=+2149.751886071" watchObservedRunningTime="2026-07-09 19:01:22.687889238 +0000 UTC m=+2149.752554911" Jul 09 19:01:25.302310 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:25.302227 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:01:28.788801 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:28.788762 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:01:31.939952 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:31.939916 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:31.940414 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:31.939965 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:31.951985 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:31.951959 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:32.717540 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:32.717505 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:01:42.173926 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:01:42.173889 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:02:00.697671 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:02:00.697640 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:02:15.495816 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:02:15.495783 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:02:34.895448 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:02:34.895412 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:02:54.196986 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:02:54.196955 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:03:13.801772 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:13.801671 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:03:31.901084 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:31.901036 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:03:36.632065 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:36.632025 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br"] Jul 09 19:03:36.707713 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:36.707678 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:03:37.627025 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:37.626992 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7"] Jul 09 19:03:38.226301 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:38.226258 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:03:41.017167 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:41.017130 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:03:50.298479 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:50.298442 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:03:51.656332 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.656291 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" podUID="95a985ae-bea3-4852-ab8a-844ce1158228" containerName="main" containerID="cri-o://e0b55783b5fb709b08ee56bbd144c945c7a7bcdb9d7566f3a82786dfb3433d46" gracePeriod=45 Jul 09 19:03:51.911471 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.911407 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:03:51.989873 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.989836 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a985ae-bea3-4852-ab8a-844ce1158228-tls-certs\") pod \"95a985ae-bea3-4852-ab8a-844ce1158228\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " Jul 09 19:03:51.989873 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.989877 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-model-cache\") pod \"95a985ae-bea3-4852-ab8a-844ce1158228\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " Jul 09 19:03:51.990118 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.989905 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-dshm\") pod \"95a985ae-bea3-4852-ab8a-844ce1158228\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " Jul 09 19:03:51.990118 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.989941 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kxm6\" (UniqueName: \"kubernetes.io/projected/95a985ae-bea3-4852-ab8a-844ce1158228-kube-api-access-9kxm6\") pod \"95a985ae-bea3-4852-ab8a-844ce1158228\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " Jul 09 19:03:51.990118 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.989970 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-tmp-dir\") pod \"95a985ae-bea3-4852-ab8a-844ce1158228\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " Jul 09 19:03:51.990118 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.990044 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-home\") pod \"95a985ae-bea3-4852-ab8a-844ce1158228\" (UID: \"95a985ae-bea3-4852-ab8a-844ce1158228\") " Jul 09 19:03:51.990313 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.990194 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-model-cache" (OuterVolumeSpecName: "model-cache") pod "95a985ae-bea3-4852-ab8a-844ce1158228" (UID: "95a985ae-bea3-4852-ab8a-844ce1158228"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:03:51.990313 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.990228 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95a985ae-bea3-4852-ab8a-844ce1158228" (UID: "95a985ae-bea3-4852-ab8a-844ce1158228"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:03:51.990433 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.990335 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-model-cache\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:03:51.990433 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.990354 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-tmp-dir\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:03:51.990433 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.990397 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-home" (OuterVolumeSpecName: "home") pod "95a985ae-bea3-4852-ab8a-844ce1158228" (UID: "95a985ae-bea3-4852-ab8a-844ce1158228"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:03:51.992214 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.992187 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a985ae-bea3-4852-ab8a-844ce1158228-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95a985ae-bea3-4852-ab8a-844ce1158228" (UID: "95a985ae-bea3-4852-ab8a-844ce1158228"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:03:51.992214 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.992190 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a985ae-bea3-4852-ab8a-844ce1158228-kube-api-access-9kxm6" (OuterVolumeSpecName: "kube-api-access-9kxm6") pod "95a985ae-bea3-4852-ab8a-844ce1158228" (UID: "95a985ae-bea3-4852-ab8a-844ce1158228"). InnerVolumeSpecName "kube-api-access-9kxm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:03:51.992463 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:51.992442 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-dshm" (OuterVolumeSpecName: "dshm") pod "95a985ae-bea3-4852-ab8a-844ce1158228" (UID: "95a985ae-bea3-4852-ab8a-844ce1158228"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:03:52.091677 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.091633 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a985ae-bea3-4852-ab8a-844ce1158228-tls-certs\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:03:52.091677 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.091661 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-dshm\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:03:52.091677 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.091670 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kxm6\" (UniqueName: \"kubernetes.io/projected/95a985ae-bea3-4852-ab8a-844ce1158228-kube-api-access-9kxm6\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:03:52.091677 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.091679 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a985ae-bea3-4852-ab8a-844ce1158228-home\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:03:52.312663 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.312575 2558 generic.go:356] "Generic (PLEG): container finished" podID="95a985ae-bea3-4852-ab8a-844ce1158228" containerID="e0b55783b5fb709b08ee56bbd144c945c7a7bcdb9d7566f3a82786dfb3433d46" exitCode=0 Jul 09 19:03:52.312663 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.312617 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" Jul 09 19:03:52.312925 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.312671 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" event={"ID":"95a985ae-bea3-4852-ab8a-844ce1158228","Type":"ContainerDied","Data":"e0b55783b5fb709b08ee56bbd144c945c7a7bcdb9d7566f3a82786dfb3433d46"} Jul 09 19:03:52.312925 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.312731 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br" event={"ID":"95a985ae-bea3-4852-ab8a-844ce1158228","Type":"ContainerDied","Data":"fea6cf82c58e974fe08899f492030771b0a93ca3394389c16568bbb1672c7b6b"} Jul 09 19:03:52.312925 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.312756 2558 scope.go:117] "RemoveContainer" containerID="e0b55783b5fb709b08ee56bbd144c945c7a7bcdb9d7566f3a82786dfb3433d46" Jul 09 19:03:52.323631 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.323597 2558 scope.go:117] "RemoveContainer" containerID="e0b55783b5fb709b08ee56bbd144c945c7a7bcdb9d7566f3a82786dfb3433d46" Jul 09 19:03:52.324012 ip-10-0-138-253 kubenswrapper[2558]: E0709 19:03:52.323985 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b55783b5fb709b08ee56bbd144c945c7a7bcdb9d7566f3a82786dfb3433d46\": container with ID starting with e0b55783b5fb709b08ee56bbd144c945c7a7bcdb9d7566f3a82786dfb3433d46 not found: ID does not exist" containerID="e0b55783b5fb709b08ee56bbd144c945c7a7bcdb9d7566f3a82786dfb3433d46" Jul 09 19:03:52.324126 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.324020 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b55783b5fb709b08ee56bbd144c945c7a7bcdb9d7566f3a82786dfb3433d46"} err="failed to get container status \"e0b55783b5fb709b08ee56bbd144c945c7a7bcdb9d7566f3a82786dfb3433d46\": rpc error: code = NotFound desc = could not find container \"e0b55783b5fb709b08ee56bbd144c945c7a7bcdb9d7566f3a82786dfb3433d46\": container with ID starting with e0b55783b5fb709b08ee56bbd144c945c7a7bcdb9d7566f3a82786dfb3433d46 not found: ID does not exist" Jul 09 19:03:52.337847 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.337822 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br"] Jul 09 19:03:52.340010 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.339984 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4805f3/auth-test-model-4805f3-kserve-796b65875f-9c5br"] Jul 09 19:03:52.647660 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.647626 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" podUID="e3de87bd-6eba-4c63-a262-0baef48813cc" containerName="main" containerID="cri-o://afd728069dce6a7ff51a093f0240afd223e3aae66624437e0d544db54c9c8811" gracePeriod=45 Jul 09 19:03:52.707172 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.707141 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" podUID="e3de87bd-6eba-4c63-a262-0baef48813cc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": dial tcp 10.134.0.37:8000: connect: connection refused" Jul 09 19:03:52.895201 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.895180 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:03:52.998473 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.998392 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-model-cache\") pod \"e3de87bd-6eba-4c63-a262-0baef48813cc\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " Jul 09 19:03:52.998473 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.998428 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jdl8\" (UniqueName: \"kubernetes.io/projected/e3de87bd-6eba-4c63-a262-0baef48813cc-kube-api-access-4jdl8\") pod \"e3de87bd-6eba-4c63-a262-0baef48813cc\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " Jul 09 19:03:52.998473 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.998464 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-dshm\") pod \"e3de87bd-6eba-4c63-a262-0baef48813cc\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " Jul 09 19:03:52.998785 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.998492 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3de87bd-6eba-4c63-a262-0baef48813cc-tls-certs\") pod \"e3de87bd-6eba-4c63-a262-0baef48813cc\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " Jul 09 19:03:52.998785 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.998526 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-home\") pod \"e3de87bd-6eba-4c63-a262-0baef48813cc\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " Jul 09 19:03:52.998785 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.998559 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-tmp-dir\") pod \"e3de87bd-6eba-4c63-a262-0baef48813cc\" (UID: \"e3de87bd-6eba-4c63-a262-0baef48813cc\") " Jul 09 19:03:52.998785 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.998664 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3de87bd-6eba-4c63-a262-0baef48813cc" (UID: "e3de87bd-6eba-4c63-a262-0baef48813cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:03:52.998959 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.998842 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-home" (OuterVolumeSpecName: "home") pod "e3de87bd-6eba-4c63-a262-0baef48813cc" (UID: "e3de87bd-6eba-4c63-a262-0baef48813cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:03:52.998959 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.998868 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3de87bd-6eba-4c63-a262-0baef48813cc" (UID: "e3de87bd-6eba-4c63-a262-0baef48813cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:03:52.998959 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:52.998896 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-model-cache\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:03:53.000668 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.000637 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-dshm" (OuterVolumeSpecName: "dshm") pod "e3de87bd-6eba-4c63-a262-0baef48813cc" (UID: "e3de87bd-6eba-4c63-a262-0baef48813cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:03:53.000791 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.000731 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3de87bd-6eba-4c63-a262-0baef48813cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3de87bd-6eba-4c63-a262-0baef48813cc" (UID: "e3de87bd-6eba-4c63-a262-0baef48813cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:03:53.000791 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.000745 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3de87bd-6eba-4c63-a262-0baef48813cc-kube-api-access-4jdl8" (OuterVolumeSpecName: "kube-api-access-4jdl8") pod "e3de87bd-6eba-4c63-a262-0baef48813cc" (UID: "e3de87bd-6eba-4c63-a262-0baef48813cc"). InnerVolumeSpecName "kube-api-access-4jdl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:03:53.099805 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.099776 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-home\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:03:53.099805 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.099800 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-tmp-dir\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:03:53.099805 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.099810 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jdl8\" (UniqueName: \"kubernetes.io/projected/e3de87bd-6eba-4c63-a262-0baef48813cc-kube-api-access-4jdl8\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:03:53.100007 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.099819 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3de87bd-6eba-4c63-a262-0baef48813cc-dshm\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:03:53.100007 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.099827 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3de87bd-6eba-4c63-a262-0baef48813cc-tls-certs\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:03:53.318415 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.318327 2558 generic.go:356] "Generic (PLEG): container finished" podID="e3de87bd-6eba-4c63-a262-0baef48813cc" containerID="afd728069dce6a7ff51a093f0240afd223e3aae66624437e0d544db54c9c8811" exitCode=0 Jul 09 19:03:53.318415 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.318381 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" Jul 09 19:03:53.318657 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.318412 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" event={"ID":"e3de87bd-6eba-4c63-a262-0baef48813cc","Type":"ContainerDied","Data":"afd728069dce6a7ff51a093f0240afd223e3aae66624437e0d544db54c9c8811"} Jul 09 19:03:53.318657 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.318454 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7" event={"ID":"e3de87bd-6eba-4c63-a262-0baef48813cc","Type":"ContainerDied","Data":"9b27f5abf1535edd64267a660d82cabf018c499909cf6a889e5bfbdd4a2d1f8e"} Jul 09 19:03:53.318657 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.318477 2558 scope.go:117] "RemoveContainer" containerID="afd728069dce6a7ff51a093f0240afd223e3aae66624437e0d544db54c9c8811" Jul 09 19:03:53.327108 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.327090 2558 scope.go:117] "RemoveContainer" containerID="afd728069dce6a7ff51a093f0240afd223e3aae66624437e0d544db54c9c8811" Jul 09 19:03:53.327366 ip-10-0-138-253 kubenswrapper[2558]: E0709 19:03:53.327352 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd728069dce6a7ff51a093f0240afd223e3aae66624437e0d544db54c9c8811\": container with ID starting with afd728069dce6a7ff51a093f0240afd223e3aae66624437e0d544db54c9c8811 not found: ID does not exist" containerID="afd728069dce6a7ff51a093f0240afd223e3aae66624437e0d544db54c9c8811" Jul 09 19:03:53.327405 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.327375 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd728069dce6a7ff51a093f0240afd223e3aae66624437e0d544db54c9c8811"} err="failed to get container status \"afd728069dce6a7ff51a093f0240afd223e3aae66624437e0d544db54c9c8811\": rpc error: code = NotFound desc = could not find container \"afd728069dce6a7ff51a093f0240afd223e3aae66624437e0d544db54c9c8811\": container with ID starting with afd728069dce6a7ff51a093f0240afd223e3aae66624437e0d544db54c9c8811 not found: ID does not exist" Jul 09 19:03:53.344673 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.344646 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7"] Jul 09 19:03:53.348890 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.348869 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/auth-test-model-172f88-kserve-69d6586999-mm4x7"] Jul 09 19:03:53.558740 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.558686 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95a985ae-bea3-4852-ab8a-844ce1158228" path="/var/lib/kubelet/pods/95a985ae-bea3-4852-ab8a-844ce1158228/volumes" Jul 09 19:03:53.559111 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:53.559098 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3de87bd-6eba-4c63-a262-0baef48813cc" path="/var/lib/kubelet/pods/e3de87bd-6eba-4c63-a262-0baef48813cc/volumes" Jul 09 19:03:57.099559 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.099522 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg"] Jul 09 19:03:57.099996 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.099868 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3de87bd-6eba-4c63-a262-0baef48813cc" containerName="main" Jul 09 19:03:57.099996 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.099880 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3de87bd-6eba-4c63-a262-0baef48813cc" containerName="main" Jul 09 19:03:57.099996 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.099899 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a985ae-bea3-4852-ab8a-844ce1158228" containerName="main" Jul 09 19:03:57.099996 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.099904 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a985ae-bea3-4852-ab8a-844ce1158228" containerName="main" Jul 09 19:03:57.099996 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.099961 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3de87bd-6eba-4c63-a262-0baef48813cc" containerName="main" Jul 09 19:03:57.099996 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.099970 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a985ae-bea3-4852-ab8a-844ce1158228" containerName="main" Jul 09 19:03:57.103963 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.103946 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.107942 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.107920 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-172f88\"/\"default-dockercfg-894hs\"" Jul 09 19:03:57.108072 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.107944 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-172f88\"/\"sub-test-model-172f88-kserve-self-signed-certs\"" Jul 09 19:03:57.108072 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.107962 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-172f88\"/\"kube-root-ca.crt\"" Jul 09 19:03:57.108072 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.108061 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-172f88\"/\"openshift-service-ca.crt\"" Jul 09 19:03:57.114806 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.114783 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg"] Jul 09 19:03:57.185960 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.185926 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:03:57.235364 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.235337 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-home\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.235465 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.235384 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-model-cache\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.235465 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.235442 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-tmp-dir\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.235543 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.235474 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-dshm\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.235577 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.235547 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmtk\" (UniqueName: \"kubernetes.io/projected/7e454334-72ca-45d9-82c8-e7422c86309d-kube-api-access-lgmtk\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.235615 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.235573 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e454334-72ca-45d9-82c8-e7422c86309d-tls-certs\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.336589 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.336566 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-home\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.336759 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.336608 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-model-cache\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.336759 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.336631 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-tmp-dir\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.336759 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.336650 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-dshm\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.336759 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.336679 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmtk\" (UniqueName: \"kubernetes.io/projected/7e454334-72ca-45d9-82c8-e7422c86309d-kube-api-access-lgmtk\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.336759 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.336728 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e454334-72ca-45d9-82c8-e7422c86309d-tls-certs\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.337000 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.336981 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-home\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.337056 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.337028 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-tmp-dir\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.337118 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.337063 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-model-cache\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.339022 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.339000 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-dshm\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.339415 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.339396 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e454334-72ca-45d9-82c8-e7422c86309d-tls-certs\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.355092 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.355040 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmtk\" (UniqueName: \"kubernetes.io/projected/7e454334-72ca-45d9-82c8-e7422c86309d-kube-api-access-lgmtk\") pod \"sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.413043 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.413018 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:03:57.546685 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:57.546663 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg"] Jul 09 19:03:57.548740 ip-10-0-138-253 kubenswrapper[2558]: W0709 19:03:57.548693 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e454334_72ca_45d9_82c8_e7422c86309d.slice/crio-e8bf4880035b929f16c6c08dc219e9cd7a032e8b7ab4f064cd3c2441578321b7 WatchSource:0}: Error finding container e8bf4880035b929f16c6c08dc219e9cd7a032e8b7ab4f064cd3c2441578321b7: Status 404 returned error can't find the container with id e8bf4880035b929f16c6c08dc219e9cd7a032e8b7ab4f064cd3c2441578321b7 Jul 09 19:03:58.349060 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:58.349028 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" event={"ID":"7e454334-72ca-45d9-82c8-e7422c86309d","Type":"ContainerStarted","Data":"ca0e40c54dc9a5bef97ef3dafa029653400e5ccedd3fb3e2ed48db5e163740c9"} Jul 09 19:03:58.349426 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:58.349070 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" event={"ID":"7e454334-72ca-45d9-82c8-e7422c86309d","Type":"ContainerStarted","Data":"e8bf4880035b929f16c6c08dc219e9cd7a032e8b7ab4f064cd3c2441578321b7"} Jul 09 19:03:58.369694 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:03:58.369645 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" podStartSLOduration=1.369629918 podStartE2EDuration="1.369629918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:58.368026272 +0000 UTC m=+2305.432691949" watchObservedRunningTime="2026-07-09 19:03:58.369629918 +0000 UTC m=+2305.434295592" Jul 09 19:04:04.997092 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:04.997046 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:04:07.413668 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:07.413634 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:04:07.414108 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:07.413694 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:04:07.426158 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:07.426129 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:04:08.405294 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:08.405262 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:04:12.450546 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:12.450504 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:04:12.556477 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:12.556445 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:04:21.076226 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:21.076189 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:04:26.691935 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:26.691853 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:04:43.559677 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:43.559645 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:04:44.598544 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:44.598510 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg"] Jul 09 19:04:44.606997 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:44.606967 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:04:48.714502 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:48.714464 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:04:52.675549 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:52.675514 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:04:59.617961 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.617892 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" podUID="7e454334-72ca-45d9-82c8-e7422c86309d" containerName="main" containerID="cri-o://ca0e40c54dc9a5bef97ef3dafa029653400e5ccedd3fb3e2ed48db5e163740c9" gracePeriod=45 Jul 09 19:04:59.862314 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.862290 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:04:59.957555 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.957524 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgmtk\" (UniqueName: \"kubernetes.io/projected/7e454334-72ca-45d9-82c8-e7422c86309d-kube-api-access-lgmtk\") pod \"7e454334-72ca-45d9-82c8-e7422c86309d\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " Jul 09 19:04:59.957765 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.957574 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-home\") pod \"7e454334-72ca-45d9-82c8-e7422c86309d\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " Jul 09 19:04:59.957765 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.957598 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-dshm\") pod \"7e454334-72ca-45d9-82c8-e7422c86309d\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " Jul 09 19:04:59.957765 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.957621 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-tmp-dir\") pod \"7e454334-72ca-45d9-82c8-e7422c86309d\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " Jul 09 19:04:59.957765 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.957739 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-model-cache\") pod \"7e454334-72ca-45d9-82c8-e7422c86309d\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " Jul 09 19:04:59.957999 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.957773 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e454334-72ca-45d9-82c8-e7422c86309d-tls-certs\") pod \"7e454334-72ca-45d9-82c8-e7422c86309d\" (UID: \"7e454334-72ca-45d9-82c8-e7422c86309d\") " Jul 09 19:04:59.957999 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.957896 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-home" (OuterVolumeSpecName: "home") pod "7e454334-72ca-45d9-82c8-e7422c86309d" (UID: "7e454334-72ca-45d9-82c8-e7422c86309d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:04:59.958105 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.958001 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e454334-72ca-45d9-82c8-e7422c86309d" (UID: "7e454334-72ca-45d9-82c8-e7422c86309d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:04:59.958105 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.958029 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e454334-72ca-45d9-82c8-e7422c86309d" (UID: "7e454334-72ca-45d9-82c8-e7422c86309d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:04:59.958105 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.958068 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-home\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:04:59.959806 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.959776 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-dshm" (OuterVolumeSpecName: "dshm") pod "7e454334-72ca-45d9-82c8-e7422c86309d" (UID: "7e454334-72ca-45d9-82c8-e7422c86309d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:04:59.959918 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.959844 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e454334-72ca-45d9-82c8-e7422c86309d-kube-api-access-lgmtk" (OuterVolumeSpecName: "kube-api-access-lgmtk") pod "7e454334-72ca-45d9-82c8-e7422c86309d" (UID: "7e454334-72ca-45d9-82c8-e7422c86309d"). InnerVolumeSpecName "kube-api-access-lgmtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:04:59.959918 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:04:59.959854 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e454334-72ca-45d9-82c8-e7422c86309d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e454334-72ca-45d9-82c8-e7422c86309d" (UID: "7e454334-72ca-45d9-82c8-e7422c86309d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:05:00.058640 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:00.058609 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgmtk\" (UniqueName: \"kubernetes.io/projected/7e454334-72ca-45d9-82c8-e7422c86309d-kube-api-access-lgmtk\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:05:00.058640 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:00.058635 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-dshm\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:05:00.058640 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:00.058645 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-tmp-dir\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:05:00.058885 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:00.058655 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e454334-72ca-45d9-82c8-e7422c86309d-model-cache\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:05:00.058885 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:00.058663 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e454334-72ca-45d9-82c8-e7422c86309d-tls-certs\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:05:00.620824 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:00.620784 2558 generic.go:356] "Generic (PLEG): container finished" podID="7e454334-72ca-45d9-82c8-e7422c86309d" containerID="ca0e40c54dc9a5bef97ef3dafa029653400e5ccedd3fb3e2ed48db5e163740c9" exitCode=0 Jul 09 19:05:00.621294 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:00.620835 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" Jul 09 19:05:00.621294 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:00.620856 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" event={"ID":"7e454334-72ca-45d9-82c8-e7422c86309d","Type":"ContainerDied","Data":"ca0e40c54dc9a5bef97ef3dafa029653400e5ccedd3fb3e2ed48db5e163740c9"} Jul 09 19:05:00.621294 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:00.620894 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg" event={"ID":"7e454334-72ca-45d9-82c8-e7422c86309d","Type":"ContainerDied","Data":"e8bf4880035b929f16c6c08dc219e9cd7a032e8b7ab4f064cd3c2441578321b7"} Jul 09 19:05:00.621294 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:00.620917 2558 scope.go:117] "RemoveContainer" containerID="ca0e40c54dc9a5bef97ef3dafa029653400e5ccedd3fb3e2ed48db5e163740c9" Jul 09 19:05:00.629667 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:00.629646 2558 scope.go:117] "RemoveContainer" containerID="ca0e40c54dc9a5bef97ef3dafa029653400e5ccedd3fb3e2ed48db5e163740c9" Jul 09 19:05:00.629957 ip-10-0-138-253 kubenswrapper[2558]: E0709 19:05:00.629930 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0e40c54dc9a5bef97ef3dafa029653400e5ccedd3fb3e2ed48db5e163740c9\": container with ID starting with ca0e40c54dc9a5bef97ef3dafa029653400e5ccedd3fb3e2ed48db5e163740c9 not found: ID does not exist" containerID="ca0e40c54dc9a5bef97ef3dafa029653400e5ccedd3fb3e2ed48db5e163740c9" Jul 09 19:05:00.630026 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:00.629964 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0e40c54dc9a5bef97ef3dafa029653400e5ccedd3fb3e2ed48db5e163740c9"} err="failed to get container status \"ca0e40c54dc9a5bef97ef3dafa029653400e5ccedd3fb3e2ed48db5e163740c9\": rpc error: code = NotFound desc = could not find container \"ca0e40c54dc9a5bef97ef3dafa029653400e5ccedd3fb3e2ed48db5e163740c9\": container with ID starting with ca0e40c54dc9a5bef97ef3dafa029653400e5ccedd3fb3e2ed48db5e163740c9 not found: ID does not exist" Jul 09 19:05:00.658092 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:00.658067 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg"] Jul 09 19:05:00.664413 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:00.664394 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/sub-test-model-172f88-kserve-6d88bd4bdc-kp6cg"] Jul 09 19:05:01.559168 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:01.559137 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e454334-72ca-45d9-82c8-e7422c86309d" path="/var/lib/kubelet/pods/7e454334-72ca-45d9-82c8-e7422c86309d/volumes" Jul 09 19:05:13.456941 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:13.456905 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:05:25.296684 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.296650 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn"] Jul 09 19:05:25.297168 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.297009 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e454334-72ca-45d9-82c8-e7422c86309d" containerName="main" Jul 09 19:05:25.297168 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.297022 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e454334-72ca-45d9-82c8-e7422c86309d" containerName="main" Jul 09 19:05:25.297168 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.297076 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e454334-72ca-45d9-82c8-e7422c86309d" containerName="main" Jul 09 19:05:25.299990 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.299974 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.303482 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.303457 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4805f3\"/\"openshift-service-ca.crt\"" Jul 09 19:05:25.303482 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.303475 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4805f3\"/\"kube-root-ca.crt\"" Jul 09 19:05:25.303669 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.303457 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4805f3\"/\"default-dockercfg-97whk\"" Jul 09 19:05:25.303669 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.303513 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4805f3\"/\"rate-test-model-4805f3-kserve-self-signed-certs\"" Jul 09 19:05:25.311096 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.311073 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn"] Jul 09 19:05:25.324971 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.324946 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:05:25.367375 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.367344 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvnfg\" (UniqueName: \"kubernetes.io/projected/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-kube-api-access-nvnfg\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.367515 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.367385 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-dshm\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.367515 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.367443 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-tls-certs\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.367515 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.367488 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-model-cache\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.367660 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.367516 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-home\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.367660 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.367534 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-tmp-dir\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.468617 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.468572 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-model-cache\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.468617 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.468624 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-home\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.468882 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.468654 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-tmp-dir\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.468882 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.468782 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvnfg\" (UniqueName: \"kubernetes.io/projected/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-kube-api-access-nvnfg\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.468882 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.468815 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-dshm\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.468882 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.468837 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-tls-certs\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.469100 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.469076 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-home\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.469154 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.469122 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-tmp-dir\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.469197 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.469142 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-model-cache\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.471013 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.470994 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-dshm\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.471265 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.471240 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-tls-certs\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.477947 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.477926 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvnfg\" (UniqueName: \"kubernetes.io/projected/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-kube-api-access-nvnfg\") pod \"rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.609683 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.609606 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:25.744396 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:25.744357 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn"] Jul 09 19:05:25.747091 ip-10-0-138-253 kubenswrapper[2558]: W0709 19:05:25.747064 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod084f39d3_6e1a_4c9b_a6a5_b35db7eff1e9.slice/crio-3f9d4cb0e345d019c526b9b6e1fb9e52ac5d35ab8219ddd97d05b12779fdbf92 WatchSource:0}: Error finding container 3f9d4cb0e345d019c526b9b6e1fb9e52ac5d35ab8219ddd97d05b12779fdbf92: Status 404 returned error can't find the container with id 3f9d4cb0e345d019c526b9b6e1fb9e52ac5d35ab8219ddd97d05b12779fdbf92 Jul 09 19:05:26.731493 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:26.731454 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" event={"ID":"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9","Type":"ContainerStarted","Data":"573481bf90e3a7d9d9f31be5829ee02aad812570844c8142aca87179a0175d2d"} Jul 09 19:05:26.731925 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:26.731506 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" event={"ID":"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9","Type":"ContainerStarted","Data":"3f9d4cb0e345d019c526b9b6e1fb9e52ac5d35ab8219ddd97d05b12779fdbf92"} Jul 09 19:05:26.751579 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:26.751526 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" podStartSLOduration=1.7515102489999999 podStartE2EDuration="1.751510249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:05:26.750411948 +0000 UTC m=+2393.815077622" watchObservedRunningTime="2026-07-09 19:05:26.751510249 +0000 UTC m=+2393.816175922" Jul 09 19:05:29.898132 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:29.898098 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:05:33.484901 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:33.484871 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 19:05:33.487209 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:33.487189 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 19:05:35.610721 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:35.610669 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:35.610721 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:35.610727 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:35.623232 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:35.623207 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:35.782212 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:35.782181 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:05:38.749481 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:38.749438 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:05:38.849526 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:38.849492 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:05:49.106140 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:49.106094 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:05:49.206526 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:49.206487 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:05:57.789815 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.789732 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7"] Jul 09 19:05:57.793052 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.793036 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.795575 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.795551 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-172f88\"/\"openshift-service-ca.crt\"" Jul 09 19:05:57.796671 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.796646 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-172f88\"/\"default-dockercfg-894hs\"" Jul 09 19:05:57.796797 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.796735 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-172f88\"/\"kube-root-ca.crt\"" Jul 09 19:05:57.796797 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.796751 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-172f88\"/\"rate-test-model-172f88-kserve-self-signed-certs\"" Jul 09 19:05:57.803893 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.803871 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7"] Jul 09 19:05:57.815046 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.815012 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:05:57.829312 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.829286 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-home\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.829397 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.829316 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-tmp-dir\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.829397 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.829347 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-dshm\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.829397 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.829364 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w98gf\" (UniqueName: \"kubernetes.io/projected/009d8f4c-9025-46e5-b6f5-c9a689853ac3-kube-api-access-w98gf\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.829503 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.829399 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009d8f4c-9025-46e5-b6f5-c9a689853ac3-tls-certs\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.829503 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.829423 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-model-cache\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.930805 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.930762 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-dshm\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.930971 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.930812 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w98gf\" (UniqueName: \"kubernetes.io/projected/009d8f4c-9025-46e5-b6f5-c9a689853ac3-kube-api-access-w98gf\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.930971 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.930874 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009d8f4c-9025-46e5-b6f5-c9a689853ac3-tls-certs\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.930971 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.930915 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-model-cache\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.930971 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.930963 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-home\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.931193 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.930989 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-tmp-dir\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.931307 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.931289 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-home\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.931374 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.931327 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-model-cache\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.931414 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.931378 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-tmp-dir\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.933221 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.933202 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-dshm\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.933320 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.933304 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009d8f4c-9025-46e5-b6f5-c9a689853ac3-tls-certs\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:57.938425 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:57.938403 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w98gf\" (UniqueName: \"kubernetes.io/projected/009d8f4c-9025-46e5-b6f5-c9a689853ac3-kube-api-access-w98gf\") pod \"rate-test-model-172f88-kserve-5574bd7675-lr8s7\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:58.101953 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:58.101864 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:05:58.229501 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:58.229469 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7"] Jul 09 19:05:58.232274 ip-10-0-138-253 kubenswrapper[2558]: W0709 19:05:58.232246 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod009d8f4c_9025_46e5_b6f5_c9a689853ac3.slice/crio-03f3a9e276218c88c844fbf187daecd2442bb3fb0c2fdfa7b4ec509fc3c1c6f3 WatchSource:0}: Error finding container 03f3a9e276218c88c844fbf187daecd2442bb3fb0c2fdfa7b4ec509fc3c1c6f3: Status 404 returned error can't find the container with id 03f3a9e276218c88c844fbf187daecd2442bb3fb0c2fdfa7b4ec509fc3c1c6f3 Jul 09 19:05:58.870798 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:58.870757 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" event={"ID":"009d8f4c-9025-46e5-b6f5-c9a689853ac3","Type":"ContainerStarted","Data":"71a405c68ab223cd4db8598b1b3c68010a68e1e0ed5dbb99961f7bb78ebbd29f"} Jul 09 19:05:58.870798 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:58.870801 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" event={"ID":"009d8f4c-9025-46e5-b6f5-c9a689853ac3","Type":"ContainerStarted","Data":"03f3a9e276218c88c844fbf187daecd2442bb3fb0c2fdfa7b4ec509fc3c1c6f3"} Jul 09 19:05:58.908642 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:05:58.908594 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" podStartSLOduration=1.908579757 podStartE2EDuration="1.908579757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:05:58.906096181 +0000 UTC m=+2425.970761855" watchObservedRunningTime="2026-07-09 19:05:58.908579757 +0000 UTC m=+2425.973245430" Jul 09 19:06:02.694530 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:02.694482 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:06:08.102039 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:08.101998 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:06:08.102516 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:08.102058 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:06:08.114192 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:08.114170 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:06:08.929077 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:08.929046 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:06:17.586261 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:17.586229 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:06:37.683062 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:37.683028 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:06:41.525820 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:41.525784 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn"] Jul 09 19:06:41.561095 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:41.561058 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:06:42.509269 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:42.509233 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7"] Jul 09 19:06:42.562469 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:42.562433 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:06:55.185207 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:55.185175 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:06:56.545361 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.545321 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" podUID="084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9" containerName="main" containerID="cri-o://573481bf90e3a7d9d9f31be5829ee02aad812570844c8142aca87179a0175d2d" gracePeriod=45 Jul 09 19:06:56.785951 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.785925 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:06:56.848164 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.848067 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-tmp-dir\") pod \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " Jul 09 19:06:56.848164 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.848116 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-dshm\") pod \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " Jul 09 19:06:56.848387 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.848184 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-model-cache\") pod \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " Jul 09 19:06:56.848387 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.848215 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvnfg\" (UniqueName: \"kubernetes.io/projected/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-kube-api-access-nvnfg\") pod \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " Jul 09 19:06:56.848387 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.848257 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-tls-certs\") pod \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " Jul 09 19:06:56.848387 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.848292 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-home\") pod \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\" (UID: \"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9\") " Jul 09 19:06:56.848568 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.848393 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9" (UID: "084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:06:56.848568 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.848462 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9" (UID: "084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:06:56.848644 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.848567 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-tmp-dir\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:06:56.848644 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.848583 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-model-cache\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:06:56.848786 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.848634 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-home" (OuterVolumeSpecName: "home") pod "084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9" (UID: "084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:06:56.850362 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.850333 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-dshm" (OuterVolumeSpecName: "dshm") pod "084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9" (UID: "084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:06:56.850362 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.850333 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9" (UID: "084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:06:56.850501 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.850373 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-kube-api-access-nvnfg" (OuterVolumeSpecName: "kube-api-access-nvnfg") pod "084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9" (UID: "084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9"). InnerVolumeSpecName "kube-api-access-nvnfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:06:56.949409 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.949369 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvnfg\" (UniqueName: \"kubernetes.io/projected/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-kube-api-access-nvnfg\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:06:56.949409 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.949401 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-tls-certs\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:06:56.949409 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.949410 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-home\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:06:56.949621 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:56.949431 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9-dshm\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:06:57.124749 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.124631 2558 generic.go:356] "Generic (PLEG): container finished" podID="084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9" containerID="573481bf90e3a7d9d9f31be5829ee02aad812570844c8142aca87179a0175d2d" exitCode=0 Jul 09 19:06:57.124749 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.124684 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" Jul 09 19:06:57.124749 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.124716 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" event={"ID":"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9","Type":"ContainerDied","Data":"573481bf90e3a7d9d9f31be5829ee02aad812570844c8142aca87179a0175d2d"} Jul 09 19:06:57.125005 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.124755 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn" event={"ID":"084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9","Type":"ContainerDied","Data":"3f9d4cb0e345d019c526b9b6e1fb9e52ac5d35ab8219ddd97d05b12779fdbf92"} Jul 09 19:06:57.125005 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.124781 2558 scope.go:117] "RemoveContainer" containerID="573481bf90e3a7d9d9f31be5829ee02aad812570844c8142aca87179a0175d2d" Jul 09 19:06:57.135863 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.135840 2558 scope.go:117] "RemoveContainer" containerID="573481bf90e3a7d9d9f31be5829ee02aad812570844c8142aca87179a0175d2d" Jul 09 19:06:57.136432 ip-10-0-138-253 kubenswrapper[2558]: E0709 19:06:57.136402 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573481bf90e3a7d9d9f31be5829ee02aad812570844c8142aca87179a0175d2d\": container with ID starting with 573481bf90e3a7d9d9f31be5829ee02aad812570844c8142aca87179a0175d2d not found: ID does not exist" containerID="573481bf90e3a7d9d9f31be5829ee02aad812570844c8142aca87179a0175d2d" Jul 09 19:06:57.136526 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.136441 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573481bf90e3a7d9d9f31be5829ee02aad812570844c8142aca87179a0175d2d"} err="failed to get container status \"573481bf90e3a7d9d9f31be5829ee02aad812570844c8142aca87179a0175d2d\": rpc error: code = NotFound desc = could not find container \"573481bf90e3a7d9d9f31be5829ee02aad812570844c8142aca87179a0175d2d\": container with ID starting with 573481bf90e3a7d9d9f31be5829ee02aad812570844c8142aca87179a0175d2d not found: ID does not exist" Jul 09 19:06:57.149229 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.149207 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn"] Jul 09 19:06:57.157565 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.157544 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4805f3/rate-test-model-4805f3-kserve-56c67ccbd7-jgvjn"] Jul 09 19:06:57.529292 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.529254 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" podUID="009d8f4c-9025-46e5-b6f5-c9a689853ac3" containerName="main" containerID="cri-o://71a405c68ab223cd4db8598b1b3c68010a68e1e0ed5dbb99961f7bb78ebbd29f" gracePeriod=45 Jul 09 19:06:57.559056 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.559025 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9" path="/var/lib/kubelet/pods/084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9/volumes" Jul 09 19:06:57.777265 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.777242 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:06:57.857280 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.857197 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w98gf\" (UniqueName: \"kubernetes.io/projected/009d8f4c-9025-46e5-b6f5-c9a689853ac3-kube-api-access-w98gf\") pod \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " Jul 09 19:06:57.857280 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.857247 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-home\") pod \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " Jul 09 19:06:57.857280 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.857276 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-model-cache\") pod \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " Jul 09 19:06:57.857535 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.857322 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009d8f4c-9025-46e5-b6f5-c9a689853ac3-tls-certs\") pod \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " Jul 09 19:06:57.857535 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.857356 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-dshm\") pod \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " Jul 09 19:06:57.857535 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.857407 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-tmp-dir\") pod \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\" (UID: \"009d8f4c-9025-46e5-b6f5-c9a689853ac3\") " Jul 09 19:06:57.857681 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.857571 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-home" (OuterVolumeSpecName: "home") pod "009d8f4c-9025-46e5-b6f5-c9a689853ac3" (UID: "009d8f4c-9025-46e5-b6f5-c9a689853ac3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:06:57.857681 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.857577 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-model-cache" (OuterVolumeSpecName: "model-cache") pod "009d8f4c-9025-46e5-b6f5-c9a689853ac3" (UID: "009d8f4c-9025-46e5-b6f5-c9a689853ac3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:06:57.857802 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.857718 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-home\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:06:57.857802 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.857738 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-model-cache\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:06:57.857802 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.857775 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "009d8f4c-9025-46e5-b6f5-c9a689853ac3" (UID: "009d8f4c-9025-46e5-b6f5-c9a689853ac3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:06:57.859290 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.859265 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009d8f4c-9025-46e5-b6f5-c9a689853ac3-kube-api-access-w98gf" (OuterVolumeSpecName: "kube-api-access-w98gf") pod "009d8f4c-9025-46e5-b6f5-c9a689853ac3" (UID: "009d8f4c-9025-46e5-b6f5-c9a689853ac3"). InnerVolumeSpecName "kube-api-access-w98gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:06:57.859893 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.859868 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-dshm" (OuterVolumeSpecName: "dshm") pod "009d8f4c-9025-46e5-b6f5-c9a689853ac3" (UID: "009d8f4c-9025-46e5-b6f5-c9a689853ac3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:06:57.859893 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.859879 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/009d8f4c-9025-46e5-b6f5-c9a689853ac3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "009d8f4c-9025-46e5-b6f5-c9a689853ac3" (UID: "009d8f4c-9025-46e5-b6f5-c9a689853ac3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:06:57.958873 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.958841 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w98gf\" (UniqueName: \"kubernetes.io/projected/009d8f4c-9025-46e5-b6f5-c9a689853ac3-kube-api-access-w98gf\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:06:57.958873 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.958868 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009d8f4c-9025-46e5-b6f5-c9a689853ac3-tls-certs\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:06:57.958873 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.958877 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-dshm\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:06:57.959096 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:57.958887 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009d8f4c-9025-46e5-b6f5-c9a689853ac3-tmp-dir\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:06:58.136349 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.136314 2558 generic.go:356] "Generic (PLEG): container finished" podID="009d8f4c-9025-46e5-b6f5-c9a689853ac3" containerID="71a405c68ab223cd4db8598b1b3c68010a68e1e0ed5dbb99961f7bb78ebbd29f" exitCode=0 Jul 09 19:06:58.136502 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.136362 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" Jul 09 19:06:58.136502 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.136366 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" event={"ID":"009d8f4c-9025-46e5-b6f5-c9a689853ac3","Type":"ContainerDied","Data":"71a405c68ab223cd4db8598b1b3c68010a68e1e0ed5dbb99961f7bb78ebbd29f"} Jul 09 19:06:58.136502 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.136404 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7" event={"ID":"009d8f4c-9025-46e5-b6f5-c9a689853ac3","Type":"ContainerDied","Data":"03f3a9e276218c88c844fbf187daecd2442bb3fb0c2fdfa7b4ec509fc3c1c6f3"} Jul 09 19:06:58.136502 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.136423 2558 scope.go:117] "RemoveContainer" containerID="71a405c68ab223cd4db8598b1b3c68010a68e1e0ed5dbb99961f7bb78ebbd29f" Jul 09 19:06:58.144994 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.144976 2558 scope.go:117] "RemoveContainer" containerID="71a405c68ab223cd4db8598b1b3c68010a68e1e0ed5dbb99961f7bb78ebbd29f" Jul 09 19:06:58.145244 ip-10-0-138-253 kubenswrapper[2558]: E0709 19:06:58.145224 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a405c68ab223cd4db8598b1b3c68010a68e1e0ed5dbb99961f7bb78ebbd29f\": container with ID starting with 71a405c68ab223cd4db8598b1b3c68010a68e1e0ed5dbb99961f7bb78ebbd29f not found: ID does not exist" containerID="71a405c68ab223cd4db8598b1b3c68010a68e1e0ed5dbb99961f7bb78ebbd29f" Jul 09 19:06:58.145283 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.145253 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a405c68ab223cd4db8598b1b3c68010a68e1e0ed5dbb99961f7bb78ebbd29f"} err="failed to get container status \"71a405c68ab223cd4db8598b1b3c68010a68e1e0ed5dbb99961f7bb78ebbd29f\": rpc error: code = NotFound desc = could not find container \"71a405c68ab223cd4db8598b1b3c68010a68e1e0ed5dbb99961f7bb78ebbd29f\": container with ID starting with 71a405c68ab223cd4db8598b1b3c68010a68e1e0ed5dbb99961f7bb78ebbd29f not found: ID does not exist" Jul 09 19:06:58.157276 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.157255 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7"] Jul 09 19:06:58.163132 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.163111 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-172f88/rate-test-model-172f88-kserve-5574bd7675-lr8s7"] Jul 09 19:06:58.499237 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.499155 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7662-139921169014624"] Jul 09 19:06:58.499481 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.499469 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9" containerName="main" Jul 09 19:06:58.499521 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.499482 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9" containerName="main" Jul 09 19:06:58.499521 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.499502 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="009d8f4c-9025-46e5-b6f5-c9a689853ac3" containerName="main" Jul 09 19:06:58.499521 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.499508 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="009d8f4c-9025-46e5-b6f5-c9a689853ac3" containerName="main" Jul 09 19:06:58.499611 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.499584 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="084f39d3-6e1a-4c9b-a6a5-b35db7eff1e9" containerName="main" Jul 09 19:06:58.499611 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.499591 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="009d8f4c-9025-46e5-b6f5-c9a689853ac3" containerName="main" Jul 09 19:06:58.503689 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.503672 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7662-139921169014624" Jul 09 19:06:58.506044 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.506026 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-p22g6\"" Jul 09 19:06:58.509690 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.509667 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7662-139921169014624"] Jul 09 19:06:58.564572 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.564542 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk6vk\" (UniqueName: \"kubernetes.io/projected/5602ec2f-5a52-4e0a-9288-7c0f174d89ed-kube-api-access-nk6vk\") pod \"test-curl-7662-139921169014624\" (UID: \"5602ec2f-5a52-4e0a-9288-7c0f174d89ed\") " pod="openshift-ingress/test-curl-7662-139921169014624" Jul 09 19:06:58.665400 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.665368 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk6vk\" (UniqueName: \"kubernetes.io/projected/5602ec2f-5a52-4e0a-9288-7c0f174d89ed-kube-api-access-nk6vk\") pod \"test-curl-7662-139921169014624\" (UID: \"5602ec2f-5a52-4e0a-9288-7c0f174d89ed\") " pod="openshift-ingress/test-curl-7662-139921169014624" Jul 09 19:06:58.674576 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.674547 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk6vk\" (UniqueName: \"kubernetes.io/projected/5602ec2f-5a52-4e0a-9288-7c0f174d89ed-kube-api-access-nk6vk\") pod \"test-curl-7662-139921169014624\" (UID: \"5602ec2f-5a52-4e0a-9288-7c0f174d89ed\") " pod="openshift-ingress/test-curl-7662-139921169014624" Jul 09 19:06:58.813692 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.813610 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7662-139921169014624" Jul 09 19:06:58.939915 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.939888 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7662-139921169014624"] Jul 09 19:06:58.942631 ip-10-0-138-253 kubenswrapper[2558]: W0709 19:06:58.942604 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5602ec2f_5a52_4e0a_9288_7c0f174d89ed.slice/crio-e0d2ba6ee92c03c8f7533ffdcfd6911e55c25434864efc9d33a83e0ec716c81b WatchSource:0}: Error finding container e0d2ba6ee92c03c8f7533ffdcfd6911e55c25434864efc9d33a83e0ec716c81b: Status 404 returned error can't find the container with id e0d2ba6ee92c03c8f7533ffdcfd6911e55c25434864efc9d33a83e0ec716c81b Jul 09 19:06:58.944295 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:58.944277 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 19:06:59.141434 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:59.141397 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7662-139921169014624" event={"ID":"5602ec2f-5a52-4e0a-9288-7c0f174d89ed","Type":"ContainerStarted","Data":"e0d2ba6ee92c03c8f7533ffdcfd6911e55c25434864efc9d33a83e0ec716c81b"} Jul 09 19:06:59.560233 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:06:59.560158 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="009d8f4c-9025-46e5-b6f5-c9a689853ac3" path="/var/lib/kubelet/pods/009d8f4c-9025-46e5-b6f5-c9a689853ac3/volumes" Jul 09 19:07:00.147413 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:00.147381 2558 generic.go:356] "Generic (PLEG): container finished" podID="5602ec2f-5a52-4e0a-9288-7c0f174d89ed" containerID="f19045a53da7a5ca6b4cf9b8d7798299a0366530ccf57970c958b1a448692f18" exitCode=0 Jul 09 19:07:00.147815 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:00.147461 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7662-139921169014624" event={"ID":"5602ec2f-5a52-4e0a-9288-7c0f174d89ed","Type":"ContainerDied","Data":"f19045a53da7a5ca6b4cf9b8d7798299a0366530ccf57970c958b1a448692f18"} Jul 09 19:07:01.282139 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:01.282116 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7662-139921169014624" Jul 09 19:07:01.305250 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:01.305221 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7662-139921169014624_5602ec2f-5a52-4e0a-9288-7c0f174d89ed/test-curl-7662-139921169014624/0.log" Jul 09 19:07:01.327569 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:01.327541 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7662-139921169014624"] Jul 09 19:07:01.331492 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:01.331469 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7662-139921169014624"] Jul 09 19:07:01.389231 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:01.389201 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk6vk\" (UniqueName: \"kubernetes.io/projected/5602ec2f-5a52-4e0a-9288-7c0f174d89ed-kube-api-access-nk6vk\") pod \"5602ec2f-5a52-4e0a-9288-7c0f174d89ed\" (UID: \"5602ec2f-5a52-4e0a-9288-7c0f174d89ed\") " Jul 09 19:07:01.391374 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:01.391347 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5602ec2f-5a52-4e0a-9288-7c0f174d89ed-kube-api-access-nk6vk" (OuterVolumeSpecName: "kube-api-access-nk6vk") pod "5602ec2f-5a52-4e0a-9288-7c0f174d89ed" (UID: "5602ec2f-5a52-4e0a-9288-7c0f174d89ed"). InnerVolumeSpecName "kube-api-access-nk6vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:07:01.490336 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:01.490265 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk6vk\" (UniqueName: \"kubernetes.io/projected/5602ec2f-5a52-4e0a-9288-7c0f174d89ed-kube-api-access-nk6vk\") on node \"ip-10-0-138-253.ec2.internal\" DevicePath \"\"" Jul 09 19:07:01.559313 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:01.559282 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5602ec2f-5a52-4e0a-9288-7c0f174d89ed" path="/var/lib/kubelet/pods/5602ec2f-5a52-4e0a-9288-7c0f174d89ed/volumes" Jul 09 19:07:01.570834 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:01.570810 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:01.964751 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:01.964690 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:02.100075 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:02.100041 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:02.157620 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:02.157595 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7662-139921169014624" Jul 09 19:07:02.157620 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:02.157617 2558 scope.go:117] "RemoveContainer" containerID="f19045a53da7a5ca6b4cf9b8d7798299a0366530ccf57970c958b1a448692f18" Jul 09 19:07:02.198911 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:02.198883 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:02.346093 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:02.345996 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:08.288525 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:08.288489 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:07:08.779692 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:08.779648 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:08.851819 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:08.851782 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:09.047266 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:09.047179 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:09.269644 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:09.269605 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:09.591630 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:09.591597 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:09.945832 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:09.945796 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:10.046550 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:10.046518 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:10.146031 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:10.145971 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:10.304481 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:10.304376 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:16.360798 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:16.360754 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:16.423818 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:16.423778 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:16.614234 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:16.614136 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:16.830728 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:16.830680 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:17.247028 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:17.246997 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f89d6df55-z5nvk_7777790b-7a7b-478b-b975-c0f16f8e7b5e/authorino/0.log" Jul 09 19:07:19.586397 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:19.586362 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vr4w6"] Jul 09 19:07:22.526730 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:22.526666 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pth4s_3fddb4de-381b-4b3a-8e5c-c7d1462f2645/manager/0.log" Jul 09 19:07:22.896412 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:22.896387 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59f7885b79-z4m6v_f9ebdde9-5ed0-4512-9e4b-da303966d500/copy-manifests/0.log" Jul 09 19:07:22.903572 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:22.903543 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59f7885b79-z4m6v_f9ebdde9-5ed0-4512-9e4b-da303966d500/manager/0.log" Jul 09 19:07:24.615419 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:24.615387 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f89d6df55-z5nvk_7777790b-7a7b-478b-b975-c0f16f8e7b5e/authorino/0.log" Jul 09 19:07:25.314609 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:25.314522 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-vr4w6_babbc9ec-7b26-4818-bc05-dff504353120/limitador/0.log" Jul 09 19:07:25.996474 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:25.996445 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-668f4b5f87-c8cgq_f5cb28e7-4f29-4d75-b227-6efd0e1db6f5/kube-auth-proxy/0.log" Jul 09 19:07:26.574801 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:26.574760 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76c7859458-twqvq_ac02fdbc-8732-4907-8446-81de5cc28330/router/0.log" Jul 09 19:07:27.329906 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:27.329869 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-l7svs_22132870-ffb8-4340-a82c-3e327a8c7f3e/main/0.log" Jul 09 19:07:27.442137 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:27.442108 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-qggk2_39f7cd41-d094-447e-b7bf-ebc21d8bce35/main/0.log" Jul 09 19:07:30.978181 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:30.978144 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9jbd2/must-gather-ltd4l"] Jul 09 19:07:30.978556 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:30.978443 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5602ec2f-5a52-4e0a-9288-7c0f174d89ed" containerName="test-curl-7662-139921169014624" Jul 09 19:07:30.978556 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:30.978454 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5602ec2f-5a52-4e0a-9288-7c0f174d89ed" containerName="test-curl-7662-139921169014624" Jul 09 19:07:30.978556 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:30.978511 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5602ec2f-5a52-4e0a-9288-7c0f174d89ed" containerName="test-curl-7662-139921169014624" Jul 09 19:07:30.981628 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:30.981612 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9jbd2/must-gather-ltd4l" Jul 09 19:07:30.984064 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:30.984041 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9jbd2\"/\"default-dockercfg-pg4rv\"" Jul 09 19:07:30.984135 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:30.984093 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9jbd2\"/\"kube-root-ca.crt\"" Jul 09 19:07:30.984254 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:30.984232 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9jbd2\"/\"openshift-service-ca.crt\"" Jul 09 19:07:30.991572 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:30.991550 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9jbd2/must-gather-ltd4l"] Jul 09 19:07:30.994660 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:30.994629 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:31.034775 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:31.034750 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zltvf\" (UniqueName: \"kubernetes.io/projected/dea41726-a86f-4954-acf1-0b61133ac8b2-kube-api-access-zltvf\") pod \"must-gather-ltd4l\" (UID: \"dea41726-a86f-4954-acf1-0b61133ac8b2\") " pod="openshift-must-gather-9jbd2/must-gather-ltd4l" Jul 09 19:07:31.034905 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:31.034812 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dea41726-a86f-4954-acf1-0b61133ac8b2-must-gather-output\") pod \"must-gather-ltd4l\" (UID: \"dea41726-a86f-4954-acf1-0b61133ac8b2\") " pod="openshift-must-gather-9jbd2/must-gather-ltd4l" Jul 09 19:07:31.096178 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:31.096146 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024 Jul 09 19:07:31.135941 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:31.135918 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zltvf\" (UniqueName: \"kubernetes.io/projected/dea41726-a86f-4954-acf1-0b61133ac8b2-kube-api-access-zltvf\") pod \"must-gather-ltd4l\" (UID: \"dea41726-a86f-4954-acf1-0b61133ac8b2\") " pod="openshift-must-gather-9jbd2/must-gather-ltd4l" Jul 09 19:07:31.136060 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:31.135974 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dea41726-a86f-4954-acf1-0b61133ac8b2-must-gather-output\") pod \"must-gather-ltd4l\" (UID: \"dea41726-a86f-4954-acf1-0b61133ac8b2\") " pod="openshift-must-gather-9jbd2/must-gather-ltd4l" Jul 09 19:07:31.136283 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:31.136269 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dea41726-a86f-4954-acf1-0b61133ac8b2-must-gather-output\") pod \"must-gather-ltd4l\" (UID: \"dea41726-a86f-4954-acf1-0b61133ac8b2\") " pod="openshift-must-gather-9jbd2/must-gather-ltd4l" Jul 09 19:07:31.144426 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:31.144400 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zltvf\" (UniqueName: \"kubernetes.io/projected/dea41726-a86f-4954-acf1-0b61133ac8b2-kube-api-access-zltvf\") pod \"must-gather-ltd4l\" (UID: \"dea41726-a86f-4954-acf1-0b61133ac8b2\") " pod="openshift-must-gather-9jbd2/must-gather-ltd4l" Jul 09 19:07:31.290829 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:31.290741 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9jbd2/must-gather-ltd4l" Jul 09 19:07:31.423169 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:31.423146 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9jbd2/must-gather-ltd4l"] Jul 09 19:07:31.425167 ip-10-0-138-253 kubenswrapper[2558]: W0709 19:07:31.425139 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddea41726_a86f_4954_acf1_0b61133ac8b2.slice/crio-0e702ce023483ae1e48d9e1e87c39046c677e9431c704464a8a2651e4485d15c WatchSource:0}: Error finding container 0e702ce023483ae1e48d9e1e87c39046c677e9431c704464a8a2651e4485d15c: Status 404 returned error can't find the container with id 0e702ce023483ae1e48d9e1e87c39046c677e9431c704464a8a2651e4485d15c Jul 09 19:07:32.286161 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:32.286132 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9jbd2/must-gather-ltd4l" event={"ID":"dea41726-a86f-4954-acf1-0b61133ac8b2","Type":"ContainerStarted","Data":"0e702ce023483ae1e48d9e1e87c39046c677e9431c704464a8a2651e4485d15c"} Jul 09 19:07:33.295229 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:33.295190 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9jbd2/must-gather-ltd4l" event={"ID":"dea41726-a86f-4954-acf1-0b61133ac8b2","Type":"ContainerStarted","Data":"96287f8eb7bf4f3c509bc265815fdea3f2dffeab0cc1e8fb020f68db4ac36aff"} Jul 09 19:07:33.295618 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:33.295247 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9jbd2/must-gather-ltd4l" event={"ID":"dea41726-a86f-4954-acf1-0b61133ac8b2","Type":"ContainerStarted","Data":"cadf0813fc302344846cabaf7a6110176b289966fab985baf24c3fb6a4be82d4"} Jul 09 19:07:33.317089 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:33.317024 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9jbd2/must-gather-ltd4l" podStartSLOduration=2.5081074770000003 podStartE2EDuration="3.317006002s" totalImagesPullingTime="808.898525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:07:33.315765604 +0000 UTC m=+2520.380431279" watchObservedRunningTime="2026-07-09 19:07:33.317006002 +0000 UTC m=+2520.381671676" Jul 09 19:07:33.935936 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:33.935902 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r7g5z_8cfe536a-207f-4ce7-8e2a-6a68d85fc813/global-pull-secret-syncer/0.log" Jul 09 19:07:34.000758 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:34.000723 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" Jul 09 19:07:34.130504 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:34.130476 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-253.ec2.internal_8c5edb6b0989376dda6e5d5712ab11e9/haproxy/0.log" Jul 09 19:07:38.409351 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:38.405775 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f89d6df55-z5nvk_7777790b-7a7b-478b-b975-c0f16f8e7b5e/authorino/0.log" Jul 09 19:07:38.780841 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:38.780761 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-vr4w6_babbc9ec-7b26-4818-bc05-dff504353120/limitador/0.log" Jul 09 19:07:41.075892 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:41.075863 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8m9g4_e74848c0-7cd6-4754-936c-e4733c3764f8/node-exporter/0.log" Jul 09 19:07:41.101668 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:41.101595 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8m9g4_e74848c0-7cd6-4754-936c-e4733c3764f8/kube-rbac-proxy/0.log" Jul 09 19:07:41.129830 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:41.129799 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8m9g4_e74848c0-7cd6-4754-936c-e4733c3764f8/init-textfile/0.log" Jul 09 19:07:41.646947 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:41.646910 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b594d457f-nkgsk_d5805c73-6290-41e0-8e37-4449efb07bce/telemeter-client/0.log" Jul 09 19:07:41.676352 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:41.676325 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b594d457f-nkgsk_d5805c73-6290-41e0-8e37-4449efb07bce/reload/0.log" Jul 09 19:07:41.703142 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:41.703112 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b594d457f-nkgsk_d5805c73-6290-41e0-8e37-4449efb07bce/kube-rbac-proxy/0.log" Jul 09 19:07:43.150909 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:43.150871 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-27t6j_995ece05-95be-45f0-acb3-1469eb38f416/networking-console-plugin/0.log" Jul 09 19:07:43.773865 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:43.773834 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/1.log" Jul 09 19:07:43.782119 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:43.782084 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-j9cfj_56d0c429-0dc1-4a9f-ae97-c002cc5a74ba/console-operator/2.log" Jul 09 19:07:43.883835 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:43.883801 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx"] Jul 09 19:07:43.891992 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:43.891963 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:43.896980 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:43.896952 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx"] Jul 09 19:07:43.973089 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:43.973046 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce6e743e-a3d3-42ef-b3b9-7748c0361453-sys\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:43.973282 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:43.973105 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce6e743e-a3d3-42ef-b3b9-7748c0361453-proc\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:43.973282 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:43.973129 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce6e743e-a3d3-42ef-b3b9-7748c0361453-lib-modules\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:43.973282 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:43.973270 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce6e743e-a3d3-42ef-b3b9-7748c0361453-podres\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:43.973458 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:43.973311 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmssv\" (UniqueName: \"kubernetes.io/projected/ce6e743e-a3d3-42ef-b3b9-7748c0361453-kube-api-access-hmssv\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:44.074069 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:44.073956 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce6e743e-a3d3-42ef-b3b9-7748c0361453-lib-modules\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:44.074069 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:44.074060 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce6e743e-a3d3-42ef-b3b9-7748c0361453-podres\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:44.074306 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:44.074108 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmssv\" (UniqueName: \"kubernetes.io/projected/ce6e743e-a3d3-42ef-b3b9-7748c0361453-kube-api-access-hmssv\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:44.074306 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:44.074132 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce6e743e-a3d3-42ef-b3b9-7748c0361453-sys\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:44.074306 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:44.074164 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce6e743e-a3d3-42ef-b3b9-7748c0361453-proc\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:44.074306 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:44.074183 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce6e743e-a3d3-42ef-b3b9-7748c0361453-lib-modules\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:44.074306 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:44.074203 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce6e743e-a3d3-42ef-b3b9-7748c0361453-podres\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:44.074306 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:44.074233 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce6e743e-a3d3-42ef-b3b9-7748c0361453-proc\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:44.074306 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:44.074266 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce6e743e-a3d3-42ef-b3b9-7748c0361453-sys\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:44.084275 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:44.084236 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmssv\" (UniqueName: \"kubernetes.io/projected/ce6e743e-a3d3-42ef-b3b9-7748c0361453-kube-api-access-hmssv\") pod \"perf-node-gather-daemonset-c8bjx\" (UID: \"ce6e743e-a3d3-42ef-b3b9-7748c0361453\") " pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:44.209312 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:44.209275 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:44.405496 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:44.405463 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx"] Jul 09 19:07:44.407597 ip-10-0-138-253 kubenswrapper[2558]: W0709 19:07:44.407564 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podce6e743e_a3d3_42ef_b3b9_7748c0361453.slice/crio-f0c62ffe6820305718c5dcdbdc77309f13b15423cf10f6ac854236b1e5404136 WatchSource:0}: Error finding container f0c62ffe6820305718c5dcdbdc77309f13b15423cf10f6ac854236b1e5404136: Status 404 returned error can't find the container with id f0c62ffe6820305718c5dcdbdc77309f13b15423cf10f6ac854236b1e5404136 Jul 09 19:07:45.395002 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:45.394959 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" event={"ID":"ce6e743e-a3d3-42ef-b3b9-7748c0361453","Type":"ContainerStarted","Data":"d2c0fa37ff7137673025fb937c743d92708fd18e45dbff2dc6c5eb0b2400ef42"} Jul 09 19:07:45.395418 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:45.395020 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" event={"ID":"ce6e743e-a3d3-42ef-b3b9-7748c0361453","Type":"ContainerStarted","Data":"f0c62ffe6820305718c5dcdbdc77309f13b15423cf10f6ac854236b1e5404136"} Jul 09 19:07:45.395418 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:45.395106 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:45.413079 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:45.413031 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" podStartSLOduration=2.41301487 podStartE2EDuration="2.41301487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:07:45.411677995 +0000 UTC m=+2532.476343670" watchObservedRunningTime="2026-07-09 19:07:45.41301487 +0000 UTC m=+2532.477680544" Jul 09 19:07:46.219692 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:46.219652 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4txx9_d89db1d7-0b3f-4d95-9c61-dd4f92a076ae/dns/0.log" Jul 09 19:07:46.253536 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:46.253504 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4txx9_d89db1d7-0b3f-4d95-9c61-dd4f92a076ae/kube-rbac-proxy/0.log" Jul 09 19:07:46.527109 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:46.527033 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tvjd5_47cb0898-a067-4eb1-afe9-e2b5413b2cee/dns-node-resolver/0.log" Jul 09 19:07:47.128761 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:47.128733 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4b29x_03116b04-5653-4707-8b77-a6c937981f49/node-ca/0.log" Jul 09 19:07:48.354195 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:48.354164 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-668f4b5f87-c8cgq_f5cb28e7-4f29-4d75-b227-6efd0e1db6f5/kube-auth-proxy/0.log" Jul 09 19:07:48.588876 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:48.588847 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76c7859458-twqvq_ac02fdbc-8732-4907-8446-81de5cc28330/router/0.log" Jul 09 19:07:49.318034 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:49.318003 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-twfp7_a95ca159-d9fe-45bf-85f5-5f6659f03627/serve-healthcheck-canary/0.log" Jul 09 19:07:50.052620 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:50.052592 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zpkd9_9f4ee0ac-335c-4580-bae3-16ac3c859dd0/kube-rbac-proxy/0.log" Jul 09 19:07:50.078237 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:50.078209 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zpkd9_9f4ee0ac-335c-4580-bae3-16ac3c859dd0/exporter/0.log" Jul 09 19:07:50.106283 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:50.106251 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zpkd9_9f4ee0ac-335c-4580-bae3-16ac3c859dd0/extractor/0.log" Jul 09 19:07:51.411127 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:51.411097 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9jbd2/perf-node-gather-daemonset-c8bjx" Jul 09 19:07:52.944864 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:52.944831 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pth4s_3fddb4de-381b-4b3a-8e5c-c7d1462f2645/manager/0.log" Jul 09 19:07:53.173509 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:53.173479 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59f7885b79-z4m6v_f9ebdde9-5ed0-4512-9e4b-da303966d500/manager/0.log" Jul 09 19:07:53.201900 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:53.201869 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59f7885b79-z4m6v_f9ebdde9-5ed0-4512-9e4b-da303966d500/copy-manifests/0.log" Jul 09 19:07:54.424098 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:54.424069 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-668f4b5f87-c8cgq_f5cb28e7-4f29-4d75-b227-6efd0e1db6f5/kube-auth-proxy/0.log" Jul 09 19:07:54.494648 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:54.494617 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-576fd7db4f-qxm52_faaf27be-52ea-45ec-88a3-fb1259d4f174/manager/0.log" Jul 09 19:07:54.605058 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:54.605024 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76c7859458-twqvq_ac02fdbc-8732-4907-8446-81de5cc28330/router/0.log" Jul 09 19:07:56.079555 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:56.079474 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-668f4b5f87-c8cgq_f5cb28e7-4f29-4d75-b227-6efd0e1db6f5/kube-auth-proxy/0.log" Jul 09 19:07:56.259165 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:56.259137 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76c7859458-twqvq_ac02fdbc-8732-4907-8446-81de5cc28330/router/0.log" Jul 09 19:07:59.733183 ip-10-0-138-253 kubenswrapper[2558]: I0709 19:07:59.733140 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wfz4_5f352f25-0ced-43ed-bac7-7579314f9f89/konnectivity-agent/0.log" limit=1024